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Personal Learning Environments (PLEs): 
Visions and Concepts 


Alexander Mikroyannidis, Sylvana Kroop, and Martin Wolpers 


Abstract Personal learning environments (PLEs) hold the potential to address the 
needs of formal and informal learners for multi-sourced content and easily 
customisable learning environments. This chapter presents an overview of the 
European project ROLE (Responsive Open Learning Environments), which spe- 
cialises in the development and evaluation of learning environments that can be 
personalised by individual learners according to their particular needs, thus 
enabling them to become self-regulated learners. 


Keywords Personal learning environment ¢ Self-regulated learning * Responsive 
open learning environment 


Introduction 


An ageing society and a flexible economy need lifelong learning more than ever, 
otherwise risking that school kids today know more than employees trained half a 
decade ago. Lifelong learning requires learners to actively control their learning 
activities while addressing the requirements imposed on them in their respective 
life contexts. Life context here can be the school, the university, the workplace, the 
hobby, etc. This leads to a shift from a centralised institutional teaching approach to 
a more learner-centred decentralised learning approach (Wilson 2005). In order to 
support this shift, learning environments must change to be more responsive and 
open, allowing effectively addressing individual needs of learners and teachers. 
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In this chapter, we will reflect on the approach of the European project ROLE 
(Responsive Open Learning Environments'). ROLE enables learners individually 
to compile their personal learning environments (PLEs) according to their partic- 
ular needs and goals. Consequently, the ROLE approach supports self-regulated 
learning (SRL) while taking into account the requirements from the roles of the 
learners and the teachers. 

The remainder of this chapter is structured as follows: Recent advances in 
personalised and SRL are introduced. Subsequently, the widget-based approach 
of the PLE is presented, along with the process of building a PLE using widgets. 
This is followed by an introduction to the ROLE project and its key innovations, 
with particular emphasis on evaluating these innovations via a number of case 
studies and test beds. An example of a recent research initiative that builds upon the 
results of the ROLE project is provided in the section that presents the weSPOT 
project. Finally, the chapter is concluded with a summary of the key ROLE 
contributions to technology-enhanced learning and an overview of what is 
presented by each chapter in the rest of this book. 


Personalised and Self-Regulated Learning 


The Learning Management System (LMS) has dominated technology-enhanced 
learning for several years. It has been widely used by academic institutions for 
delivering their distance learning programmes, as well as for supporting their 
students outside the classroom. The LMS has been a powerful tool in the hands 
of educators, enabling them to complement face-to-face teaching in the classroom 
with remote work by individual students, as well as groups of them (Bri et al. 2009; 
Wainwright et al. 2007; Abel 2006; Watson et al. 2007). 

However, the advent of Web 2.0 has altered the landscape in technology- 
enhanced learning. Learners nowadays have access to a variety of learning tools 
and services on the cloud. These tools and services are usually provided by different 
vendors and in many cases are open and free. However, augmenting and configur- 
ing these diverse and distributed tools and services in order to address the needs and 
preferences of individual learners is a significant challenge for modern online 
learning environments. 

This ongoing transition from the traditional approach of the LMS towards Web 
2.0-based learning solutions bears significant benefits for learners. It puts emphasis 
on their needs and preferences, providing them with a wider choice of learning 
resources to choose from. Learners usually switch learning contexts continuously, 
adapting to the respective needs automatically. The LMS is not able to provide 
learners with the required flexibility. Furthermore, the LMS is a closed system that 
does not allow the learner to take her achievements with her when changing the 
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LMS-providing learning organisation, e.g. while starting a new job, the previously 
used LMS-profile cannot easily be transferred to the new one used at the workplace. 

The PLE is a facility for an individual to access, aggregate, manipulate, and 
share digital artefacts of their ongoing learning experiences. The PLE follows a 
learner-centric approach, allowing the use of lightweight services and tools that 
belong to and are controlled by individual learners. Rather than integrating different 
services into a centralised system, the PLE provides learners with a variety of 
services and hands over control to them to select and use these services the way they 
deem fit (Chatti et al. 2007; Wilson 2008). 

The emergence of the PLE has greatly facilitated the use and sharing of open and 
reusable learning resources online. Learners can access, download, remix, and 
republish a wide variety of learning materials through open services provided on 
the cloud. Open Educational Resources (OER) can be described as “teaching, 
learning and research resources that reside in the public domain or have been 
released under an intellectual property license that permits their free use or 
repurposing by others depending on which Creative Commons license is used” 
(Atkins et al. 2007). 

SRL comprises an essential aspect of the PLE, as it enables learners to become 
“metacognitively, motivationally, and behaviourally active participants in their 
own learning process” (Zimmerman 1989). Although the psycho-pedagogical the- 
ories around SRL predate very much the advent of the PLE, SRL is a core 
characteristic of the latter. SRL is enabled within the PLE through the learner- 
driven assembly of independent resources in a way that fulfils a specific learning 
goal. By following this paradigm, the PLE allows learners to regulate their own 
learning, thus greatly enhancing their learning outcomes (Steffens 2006; Fruhmann 
et al. 2010). 


The Widget-Based Approach of the PLE 


As online learners become more discerning in terms of the choices related to the 
types and styles of their potential study materials, they will potentially seek content 
from multiple sources. In addition, because of the flexibility and ease of use that 
enables many users to customise that content, those same learners may wish to 
personalise their learning environment. 

The PLEs presented in this book are primarily enabled by widgets, which are 
micro-applications performing a dedicated task. This task can be as simple as 
showing news headlines or weather forecasts, but also more complex like facilitat- 
ing language learning or collaborative authoring. Widgets can be either desktop 
based or web based. Desktop-based widgets reside locally on your computer and 
may access the web for information, such as a desktop widget that shows the local 
temperature and weather. Web-based widgets reside on the web and can be embed- 
ded on a web page, such as an RSS reader widget that fetches news on your start 
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Fig. 1 Browser- and widget-based PLE concept 


page. Web-based widgets have proven quite popular as they enhance the interac- 
tivity and personalisation of web sites. 

As already mentioned above, the theoretical idea of PLEs is not a specific 
software application. A PLE is instead a concept based on the idea to have 
learner-centric Web 2.0-based environments individually designed. It is not a 
one-size-fits-all learning environment but a personalised environment a learner 
takes control over his/her own learning process instead of being controlled by a 
pre-delivered orchestration of learning goals, tools, services, and content. In PLE 
research it is deemed essential to have a learner challenged by offering her the 
ability to create her individually controlled and preferred learning environment. 

In the ROLE project, the basic equipment for creating PLEs has been developed 
according to the idea of an easy drag-and-drop system of widgets. 

On the one hand, a repository (widget store) is necessary to store and adminis- 
trate useful widgets. On the other hand an enabler space (widget space) is necessary 
to have learners their individually preferred widgets integrated, used, and managed 
in their personal style. Figure 1 outlines this approach schematically. 

From a user perspective, ROLE is Software as a Service (SaaS) (Mell and 
Grance 2011; Vaquero et al. 2008)—the user does not install and run it locally. 
This paradigm affects three main aspects of the user-visible parts of ROLE: 


e Widget space: The widget space contains a number of personally selected 
learning widgets whereby all of them access and use already existing and 
established external OER. It is the virtual environment where the user installs 
and uses its widgets. 

« Single widget: A single widget abstracts (accesses and uses) at least one single 
external resource. There are widgets accessing and using just one single external 
resource, e.g. a Wikipedia widget or an LEO dictionary widget. Furthermore, 
some widgets have been implemented to make use of cloud computing to an 
extensive degree. One example is the “ROLE translator widget” which accesses 
and displays the results of different popular resources such as LEO.org, dict.cc, 
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Wikipedia, and Google translator. The results of translating a specific term are 
used from all translating resources at the same time and are displayed in the same 
place by using the ROLE translator widget. Thus a learner has a better and more 
critical overview by being able to quickly compare the provided Web 2.0-based 
translation data. One more interesting example of a cloud computing-based 
widget is “Binocs” which displays search results by using different external 
resources and depending on the used resources of a personal network of trusted 
friends, colleagues, and experts (Govaerts et al. 2011). All widgets can be found 
in the ROLE Widget Store described in the following section. 

¢ Multiple devices: ROLE widgets and content can be accessed and used with 
different devices. Depending on the widgets and content, it can be used by all 
kind of browser-based applications on notebooks, smartphones, tablets, etc. 


Building a Widget-Based PLE 


In order to build a widget-based PLE, the learner will need to access a widget store. 
A widget store is a directory of widgets, where widgets are commonly categorised 
according to their purpose, e.g. widgets for planning, communication, and collab- 
oration. An example of such a categorisation is shown in Fig. 2. The learner can 
browse and download the widgets, as well as provide feedback on them in the form 
of ratings and comments. The ROLE project has built a widget store dedicated to 
widgets for learning purposes.” 

After selecting the appropriate widgets, the learner needs to add them to the web 
space of their choice and start using them for their learning, either by themselves or 
in collaboration with other learners. Widgets can also be embedded inside an LMS, 
such as Moodle, thus enhancing its functionality and content, as shown in Fig. 3. 
Additionally, ROLE offers a facility for creating a shared learning space and 
populating it with widgets.’ 

For more information on building a PLE and using it to become a self-regulated 
learner, one can refer to the following free online courses that have been developed 
by ROLE: 


¢ Responsive open learning environments": This course provides an overview of 
the concepts behind PLEs and also demonstrates a selection of ROLE widgets 
within learning activities. Figure 4 shows such an activity, where the learner is 
invited to use a ROLE widget in order to complete a series of learning tasks. 


2 http://www.role-widgetstore.eu 
3 http://role-sandbox.eu 
4 http://labspace.open.ac.uk/course/view.php?id=7433 
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Tools, or widgets, are smal web based software applicators supporting specic leamng oF teaching goals and tasks. Widgets can be added to a Persona! 
Leaming Environment or a Personal Learning Management System, grouped within leaming spaces according to ther functonsities and purposes, and 
USOC to SPPON particular learning activities, C.G.: 


planning and ceganization of learning processes p reahtime leamer communication and collaboration 
‘> search for leaning content (texts, presentations, videos, etc) > reflection and evaluaton processes 
> recommendation (learning content, peers, tools, etc.) > creation and manipulation of learning content 
> taining and testing actwtes (e.g. Vocabulary Trainer) > pedagogical processes (e.g. task and activity tracking) 


Add a new tool 


Here you can add a new tool to the Widget Store. The Widget Store supports OpenSocial anc W3C widget specifications. To submi a tool you must have 
been registered at the store. You can get an account here. You can aiso add tools without being the author, if this is allowed by the given licence. After 
you submitted a too! a member of the ROLE consortium wil review your tec! and either publish it or contact you because of possible improvement. You 
can see the status of the tool at the My Content page. 


Fig. 2 The ROLE widget store offers widgets (tools) for a variety of learning purposes, 
categorised according to learning tasks 
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Fig. 3 ROLE widgets embedded inside a Moodle course. The learner uses them to search for 


learning resources, as well as collaborate with other learners through videoconferencing and a 
shared writing pad 


e Self Regulated Learning? : This course introduces the concept of SRL and guides 
learners into using the ROLE tools in order to apply the SRL principles into their 
own learning. 


The content of these two courses is also available as a free interactive eBook, 
developed for the iPad and MacOS (Mikroyannidis et al. 2013a). The eBook 
provides an introduction to the new learning technologies that empower SRL and 
PLEs. A selection of widgets that will help readers build their own PLE and become 
a self-regulated learner are also demonstrated. Readers have an opportunity to 


interact with these widgets through a set of learning activities included in the 
eBook. 


5 http://labspace.open.ac.uk/course/view.php?id=7898 


Learning Tools... &) 


C) using Leeming Toots 


Boatiine 


Binocs 
D Activity 1: Search for 
OER 


3. Building a PLE 
(8) Conclusion & Bibliography 


A. Mikroyannidis et al. 


Rar er @oear 


Responsive Open Learning Environments 
LabSpace > All Units > ProjectSpace > ROLE_1 > Activity 1: Search for OER ? 


<Social search widget Bibliography widget> 


Tn this activity, you will have the opportunity to use the Binocs widget for a particular task and additionally be 
able to rate the results of your search. 


Find the media you are looking for! 
Search in many databases at once! 


Once the results are available, you can preview the media by 
clicking ©, sharing it with your frierids using Ù and you can 
pals the nsari @ and *. 


You can finetune the mediatypes and databases, by clicking 
on “Sat the top: 


Ques Gasgen powered by Googie 


Initially, you are requested to camy out a specific search for Open Educational Resources (OER) using a term 


Table 1: Some ideas for appropriate search terms 


ae 


Fig. 4 A learning activity of a free online course developed by ROLE, featuring a search widget 
and a step-by-step guide on completing a series of learning tasks 


The ROLE Project and Its Key Innovations 


ROLE is a European-funded initiative with 16 internationally renowned research 
groups from six European countries and China. It started on February 2009 with a 
duration of 4 years. ROLE was established to research and explore a variety of tools 
and services that enable learners to build their PLE, based on their needs and 
preferences. ROLE has brought forward the innovations in PLEs and SRL, through 
research in the following directions: 


e User-centric approach to learning environments with a focus on end-user devel- 
opment to design and control a PLE. 
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e Contemporary pedagogical models for personalisation in learning networks, 
SRL and collaboration in networked communities. 

e Contemporary engineering frameworks for designing, integrating, orchestrating, 
and evaluating learning services, tools, and content. 

e Frameworks for evaluating learner interactions in learning networks. 


The notion of lifelong learning as discussed today formulates a number of 
requirements on the technological basis as well as the associated learning and 
business/organisational processes. As our target group ranges from all possible 
domains and roles, e.g. learners, teachers, companies, employer, employees, and 
learning organisations, opportunities arise that will support the current shift in 
education towards more self-regulated learners (Van Harleman 2006) in scenarios, 
where the teacher role shifts more towards a mentoring role: the centralised 
institutional teaching approach shifts to a more learner-centred decentralised learn- 
ing approach (Wilson 2005). 

The ROLE project provides solutions to this set of complex challenges by 
advancing the state of the art in the technology and methodology. The following 
sections outline the ROLE approach in technology and methodology. 

Technology: interoperable infrastructure enables PLE composition—ROLE has 
provided an infrastructure that enables learners to create their own PLEs, while 
maintaining a close link to the rules and restrictions of the education-providing 
organisation (Isaksson 2013; Dahrendorf 2013). In essence, the idea is to loosen the 
control on the learner while maintaining the ability to certify learner achievements. 
For example, the learner chooses the required learning tools and contents from a 
wide selection and compiles them into her individual PLE. At the same time, the 
education provider can control which tools and contents can be chosen by the 
learner. 

ROLE tools and content within the PLE are able to communicate with each other 
in order to enable tools and contents to react to each other based on the user 
interaction. Finally, rather than replacing LMS, the ROLE approach allows the 
successful augmentation of existing learning environments. This way, the costs for 
introducing the ROLE approach to existing learning environments is significantly 
reduced, which fosters its uptake. 

Methodology: self-regulation as the key learning paradigm—Learners today are 
not aware of the advanced learning paradigm of SRL. In most cases, the basic 
components of SRL, that is, cognition, meta-cognition, motivation, affects, and 
volition (Efklides 2009), are used by learners intuitively without understanding the 
conceptual background. Apart from supporting SRL in PLE creation and use 
through respective recommenders, collaboration tools and best practice sharing, 
ROLE raises awareness through a number of dedicated learning resources. These 
range from short videos explaining the SRL principles (see for example http:// 
youtu.be/jTalvOH6JjA), to bespoke online courses about SRL that help teachers 
and students understand the mechanics and benefits behind SRL, such as the ones 
introduced in the previous section of this chapter. 
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The ROLE innovations in technology and methodology have been proven success- 
ful in a number of case studies that investigate the impact of PLEs on different 
forms of learning. Each case study has employed large test beds that have run 
continuously throughout the lifetime of the project and beyond (Mikroyannidis and 
Connolly 2012, 2013). The ROLE test beds cover a wide variety of rich contexts 
inside and outside of Europe, in which there is potential for significant impacts on 
both personalised and SRL. Each test bed concentrates on researching a large 
sample of representative individuals; this has enabled ROLE as a whole to collect 
experiences covering a large variety of learning contexts and requirements. The 
ROLE case studies and test beds are presented in detail in subsequent chapters of 
this book. A brief overview is provided below: 

Case Study 1: Using Widget Bundles for Formal Learning in Higher Educa- 
tion—This case study explores the usefulness of the PLE for facilitating and 
complementing the learning that happens inside the classroom in Higher Education. 
The test beds of this case study are three universities: the RWTH Aachen University 
(RWTH) in Germany, the Shanghai Jiao Tong University (SJTU) in China and the 
University of Uppsala in Sweden. These test beds present a variety of learning 
cultures, different profiles of students, as well as different methods of teaching and 
learning, e.g. synchronous versus distant learning. Bespoke widget bundles were 
developed by the ROLE project in order to address the learning contexts and 
requirements of these test beds. These widget bundles were employed in different 
learning domains to support different types of learning in these test beds and were 
evaluated by both the teachers and the students. 

Case Study 2: Designing PLE for Higher Education—In this case study, the 
potential benefits associated with enabling teachers and students to design, build, 
and use their PLEs collaboratively are investigated. The test beds of this case study 
are the Tongji University in China and three Swiss universities, namely the Uni- 
versity of Fribourg, the University of Geneva, and the EPFL. In these test beds, a 
Web 2.0 platform enabling the construction, the sharing, and the repurposing of 
PLEs has been introduced. Participatory design and validation activities have been 
carried out in the framework of Higher Education, aiming at understanding the 
benefits of PLEs in academic institutions. 

Case Study 3: Exploring OER for Informal Learning—This case study concerns 
the learner’s potential transition from formal to informal learning. The test bed of 
this case study is OpenLearn,° an OER repository offered by the Open University in 
the UK. OpenLearn users are primarily informal learners, who want to find and 
study OER either individually or in collaboration with others. The ROLE interven- 
tion in the OpenLearn test bed has been about improving the informal learning 
experience in a number of ways. First of all, by enabling individuals to build and 
personalise their learning environment, thus gaining more control over the use and 
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manipulation of study materials. Additionally, the adoption of certain ROLE tools 
inside OpenLearn is offering further value to learners through fostering learning 
communities. This presents an opportunity to individual informal learners to be part 
of a shared learning experience instead of a lone study. 

Case Study 4: Technology Enhanced Workplace Learning: Learning in the 
workplace is targeted by this case study, which explores the challenges and 
opportunities associated with SRL in the workplace and the sharing of best prac- 
tices among employees. The test bed in question is Festo Lernzentrum Saar GmbH 
in Germany. Festo has experimented with the notion of the Personal Learning 
Management System (PLMS), a crossover between the PLE and the LMS. The 
PLMS aggregates learning resources and applications available in the web and 
selected by the learner. It facilitates learners in planning their learning activities, 
searching for learning content and tools, training and testing, as well as in reflecting 
and evaluating their learning progress. 

The evaluation results from the ROLE test beds are presented in detail in the 
chapters of this book that discuss each case study. Overall, the evaluation results 
indicate the best suitability of the ROLE approach for self-regulated learners while 
providing significant improvements even in traditional learning scenarios where 
ROLE tools are used for homework-like assignments. Additionally, the successful 
evaluation of the ROLE approach has led partners to include it in their commercial 
products and consulting practices. 


PLEs for Inquiry-Based Learning: The weSPOT Project 


The ROLE project has been a pioneering initiative in PLE research. It has paved the 
way for more national and international research initiatives that explore the poten- 
tial applications and benefits of PLEs in different learning contexts. Both the 
theoretical and technological frameworks developed by ROLE have been taken 
upon and extended by recent research projects. A prominent example of such an 
initiative is the weSPOT project, which is investigating the potential impact of 
PLEs in Inquiry-Based Learning. 

Inquiry-based learning (IBL) enables learners to take the role of an explorer and 
a scientist as they try to solve issues they came across and that made them wonder, 
thus tapping into their personal feelings of curiosity. IBL leads to structured 
knowledge about a domain and to more skills and competences about how to 
carry out efficient and communicable research. 

The European project weSPOT’ adopts a PLE-based approach in order to 
support learners and educators in IBL (Mikroyannidis et al. 2013c). The project 
focuses on IBL with a theoretically sound and technology-supported personal 
inquiry approach. weSPOT supports the meaningful contextualisation of scientific 
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concepts by relating them to personal curiosity, experiences, and reasoning. 
weSPOT addresses several challenges in the area of science education and tech- 
nology support for building personal conceptual knowledge (Mikroyannidis 
et al. 2013b). 

These principles have driven the development of the weSPOT inquiry space,® a 
personal and social IBL environment that reuses and extends the Elgg open-source 
social networking framework.’ The weSPOT inquiry space has been built based on 
the following requirements (Mikroyannidis 2014): 


e A widget-based architecture enables the personalisation of the inquiry environ- 
ment, allowing teachers and students to build their inquiries out of mashups of 
inquiry components. 

e Students can connect with their peers and form groups in order to build, share, 
and perform inquiries collaboratively. 


Inquiries in the weSPOT inquiry space are consistent with the weSPOT peda- 
gogical IBL model (Protopsaltis 2013). According to this model, an inquiry consists 
of the following six phases: (1) question/hypothesis, (2) operationalisation, (3) data 
collection, (4) data analysis, (5) interpretation/discussion, and (6) communication. 

The weSPOT inquiry space enables its users (teachers and students) to create 
mashups of their preferred inquiry components, assign them to different phases of 
an inquiry, share them with other users, and use them collaboratively in order to 
carry out an inquiry. When creating a new inquiry, users are provided with a set of 
recommended inquiry components for each phase of the inquiry. They can then 
customise these sets of components by adding, removing, and arranging inquiry 
components for each phase of the inquiry. 

The weSPOT inquiry space offers a variety of inquiry components to teachers 
and students, enabling them to create, edit, and share hypotheses, questions, 
answers, notes, reflections, mind maps, etc. Some of these components communi- 
cate with the APIs of REST web services offered by external tools. One of these 
external tools is the ARLearn mobile app,'° which allows students to collect 
different types of data (photos, videos, measurements, etc.) with their smartphones 
and share them with other inquiry members via the weSPOT inquiry space. A 
Learning Analytics dashboard visualises all the activities taking place within an 
inquiry, enabling teachers to monitor the progress of their students and students to 
self-monitor their progress. 

Figure 5 shows an example mashup of inquiry components for a particular phase 
of an inquiry that explores the everyday uses of batteries. The phase is labelled 
“Discuss the findings” and corresponds to the “Interpretation/Discussion” phase of 
the weSPOT IBL model. In this phase, the members of the inquiry use collabora- 
tively three inquiry components in order to discuss and interpret their findings. 


8 http://inquiry.wespot.net 
? http://elgg.org 
10 http://portal.ou.nl/en/web/arlearn 
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Fig.5 A mashup of inquiry components for discussing and interpreting the findings of an inquiry 


They use the “Inquiry discussion” component to exchange their views asynchro- 
nously in discussion forums. They also use the “Questions” component in order to 
provide answers to the key research questions of this inquiry and vote for the best 
answers. Finally, they create and share mind maps containing interpretations of 
their findings via the “Mind maps” component. 
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Additionally, users have access to external resources and widgets and can use 
them in their mashups together with the inquiry components offered by the 
weSPOT inquiry space. These resources and widgets originate from external 
LMSs, such as Moodle or Blackboard, or from external repositories of widgets, 
such as the one offered by the European project Go-Lab.'' In order to integrate 
external resources originating from LMSs, we have implemented the IMS Learning 
Tools Interoperability (LTI) specification,” thus allowing teachers to include in 
their inquiries either course components from LMSs, such as discussion forums or 
quizzes, or entire LMS courses. 


Conclusions and Book Overview 


In summary, the vision of the ROLE project has been to provide the necessary 
infrastructure and processes for any learner across the world to assemble their own 
PLE, while enabling the education provider to exercise the necessary control to 
facilitate the certification of the learning achievements. From a technical point of 
view, the approach taken by ROLE enables the flexible composition of technologies 
by the end user in the sense of mashing-up learning tools and technologies at the 
“clients” side. 

Today’s rapidly changing education and employment conditions demand a 
lifelong learner who is flexible, motivated, and in control of his or her learning. 
The ROLE initiative has significantly advanced the state of the art in technology- 
enhanced learning, by providing lifelong learners with the tools and support they 
need for personalising their learning and developing a wide range of SRL skills. 

The rest of this book presents in detail the theoretical and technological advances 
of the ROLE project, with particular emphasis on the applications of these advances 
in the case studies investigated by the project, as well as the evaluation results 
obtained by the project’s test beds. More specifically, Chap. 2 introduces the SRL 
approach of the project, consisting of a formal framework that describes the SRL 
process, related competences, and guidelines. Additionally, the methods offered by 
the ROLE SRL framework on supporting learners in order to learn in a self- 
regulated way are presented. 

Chapter 3 presents the evaluation framework and methodology of the project. 
The ROLE evaluation framework is case study-based and follows a multi-method 
approach. It integrates technological, organisational, psycho-pedagogical, and 
social aspects. At the same time, it provides a flexible and adaptive methodology, 
capable of accommodating the changes that are inevitable in the emerging field of 
personalised and SRL. 


M http://www.golabz.eu/apps 
12 http://www.imsglobal.org/lti/index.html 
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Chapters 4—7 present the project’s case studies and the test beds employed in 
each case study. As mentioned earlier in this chapter, the ROLE case studies and 
test beds cover a wide variety of learning contexts, ranging from formal to informal 
learning and from higher education to distance learning and workplace learning. 
Each test bed is presented according to its specific learning scenarios and require- 
ments, the stakeholders involved, as well as the PLE intervention achieved by the 
project and the evaluation results. 

Finally, Chap. 8 discusses the lessons learned from the development of the 
project’s PLE framework. In particular, the overall architecture and its components, 
as well as the platforms in which the ROLE technological framework has been 
integrated are presented. In addition, the experiences and lessons learned from the 
design and development of the ROLE technological framework are discussed, 
together with the lessons learned from the collaboration both internally within the 
ROLE development team and externally with other open-source projects. 
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Supporting Self-Regulated Learning 


Alexander Nussbaumer, Ingo Dahn, Sylvana Kroop, 
Alexander Mikroyannidis, and Dietrich Albert 


Abstract Self-regulated learning (SRL) competences are crucial for lifelong learn- 
ing. Their cultivation requires the right balance between freedom and guidance 
during the learning processes. Current learning systems and approaches, such as 
personal learning environments, give overwhelming freedom, but also let weak 
learners alone. Other systems, such as learning management systems or adaptive 
systems, tend to institutionalise learners too much, which does not support the 
development of SRL competences. This chapter presents possibilities and 
approaches to support SRL by the use of technology. After discussing the theo- 
retical background of SRL and related technologies, a formal framework is 
presented that describes the SRL process, related competences, and guidelines. 
Furthermore, a variety of methods is presented, how learners can be supported to 
learn in a self-regulated way. 
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Introduction 


The ROLE project’ has aimed to achieve progress beyond the state of the art in 
user-centric responsive and open learning environments, which included psycho- 
pedagogical models beyond instructional design. ROLE was researching and 
developing a psycho-pedagogical framework for supporting the individual compo- 
sition and usage of learning environments. The most important goal of this frame- 
work is to support learners to learn in a self-regulated way in responsive open 
learning environments. 

The self-regulated way of learning” is a fundamental aim for ROLE. The reason 
for this aim is the orientation towards lifelong learning, which means that learning 
with ROLE takes place across institutional boundaries. This includes consideration 
of formal, non-formal, as well as informal learning. These dimensions of learning 
organisations are seen as fluent, rather than as rigid categories. 

In order to manage the challenge of learning on their own in the ROLE context, 
learners require certain self-regulated learning (SRL) competences. For this reason, 
ROLE aims at enhancing this kind of competences by providing support strategies 
on different levels depending on the learner’s needs and competences. Basically the 
support strategy depends on the learning situation. In a blended learning situation a 
teacher or tutor plays an important role for supporting SRL, in a collaborative 
situation peers play a role and have influence on the learning trajectory, and finally 
technology-mediated approaches play a role in all learning situations. All of them 
can be interconnected and the support strategy depends on the respective situations. 

This chapter is widely based on ROLE deliverables, especially Deliverable D6.1 
(Nussbaumer et al. 2013) has been taken into account. The remainder of this chapter 
is structured as follows: Section “Theoretical Background” gives an overview on 
the theoretical background of SRL. Learning technology and its relation to SRL is 
described in section “Related Work and Technology”. Section “Models and Frame- 
work” presents the general framework on SRL and includes several models. A 
variety of guidance strategies are listed in section “Support Strategies”. The experi- 
ences made at the ROLE test beds and in several workshops with students and 
researchers are reported in section “Evaluation Results, Challenges, and Barriers”. 
Finally, the conclusion section summarises the key findings. 


Theoretical Background 


From a psycho-pedagogical point of view, SRL is a complex field of research that 
combines motivational as well as cognitive and personality theories. Components of 
SRL are cognition, meta-cognition, motivation, affects, and volition (Kitsantas 2002). 


http://www.role-project.eu/ 
? See self-regulated learning teaser video, http://www.youtube.com/watch?v=jTalvOH6JjA. 
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According to Zimmerman (2002), students can be described as self-regulated to the 
degree that they are meta-cognitively, motivationally, and behaviourally active par- 
ticipants in their own learning process. To define students’ learning as self-regulated, 
they have to use specific strategies for attaining their goals and all this has to be based 
on self-efficacy perceptions. In this context there are mainly three elements important, 
namely the SRL strategies of students, their perceptions of self-efficacy regarding 
their performance skills, and the commitment to their goals. The learners are active 
and able to control, monitor, and regulate their cognition, motivational state, behav- 
iour, and context. Furthermore, the learners set goals and try to achieve them through 
progress-monitoring. These self-regulatory activities are mediators between personal 
characteristics, contextual features, and actual performance in the learning process. In 
a meta-analysis conducted by Hattie (2009), it turned out that performing self- 
regulatory activities in the learning process is one of the most effective methods to 
reach the learning goals. 

Zimmerman has developed a cyclic SRL model (Zimmerman 2002) consisting of 
three phases, which are the forethought phase (goal setting or planning), the 
performance phase (self-observation processes), and the self-reflection phase 
(self-reflection processes). According to this model, learning performance and 
behaviour consist of both cognitive and meta-cognitive activities. The cognitive 
activities are related to dealing with subject domains, for example, acquiring 
domain knowledge through reading. The meta-cognitive activities are related to 
thinking about and regulating the cognitive activities, for example, making a plan 
about domain knowledge acquisition. 

A similar approach is pursued by Boekaerts (1999) who developed the layered 
SRL model consisting of three layers. The first layer is about the regulation of the 
self, which is related to the choice of goals and resources. The second layer focuses 
on the regulation of the learning process that relates to the use of meta-cognitive 
skills to direct the learning process. The third layer describes the regulation of the 
processing modes, which describes the choice of cognitive strategies. Also this 
model deals with cognitive and meta-cognitive activities, as well as with goals and 
resources. 

A key role in SRL is given to learning activities that are also called learning 
strategies or learning processes. Dabbagh and Kitsantas (2004) listed six key 
processes that are essential for SRL, namely goal setting, self-monitoring, self- 
evaluation, task strategies, help-seeking, and time management. 


¢ The goal setting process is defined as the outcome of a learning process and 
identifies strategies how to reach these goals. Goal setting motivates the 
learner’s choice of and attention to the relevant tasks and it also motivates to 
attain higher effort and higher persistence over the course of time (Zimmerman 
2002). Furthermore, goal setting influences learning through affective reactions, 
for example, higher self-satisfaction when goals are reached. Also the difficulty 
of a goal is an important factor for performance that increases with the difficulty 
level of the goal (Locke and Latham 2002). 
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e Self-monitoring is defined as one’s reflected attention to an aspect of behaviour 
that directs the learners’ attention to the task and assists them in evaluating the 
outcomes of their efforts. Self-monitoring is important because it helps learners 
attaining their goals by defining adequate learning adjustments. 

¢ Self-evaluation is the process where the learner compares the learning outcome 
with their own goals. It fosters better skill acquisition, self-efficacy beliefs, 
intrinsic interest, and self-satisfaction about performance. 

¢ Task strategies is defined as the process of the learner who applies strategies 
which help reach their own goals. Studies indicate that students who applied 
strategies for learning had a better performance than students who did not apply 
them as much (Pintrich 1990). 

e Help-seeking is taking place if a learner identifies and calls upon outside 
resources. Thereby not only human help is meant, but also external analogue 
and digital resources. 

+ Time management is the process where learners manage the learning regarding 
time. Effective time budgeting highly correlates with academic achievement. 


According to Roberts and Erdos (1993), meta-cognition is a key concept in the 
study of cognition and it plays an important role in the transfer of cognitive skills 
and in problem solving. Meta-cognition refers to knowledge and cognition about 
one’s own cognition. According to Treier (2004), meta-cognition is a kind of self- 
monitoring, self-observation, and self-regulation related to cognitive and infor- 
mation processing. Meta-cognition is the competence of reflecting a mental task 
critically and to organise involved learning and thinking processes in an efficient 
and effective way. The usage of meta-cognitive learning strategies is an essential 
component of SRL and is very important for flexibilisation and personalisation. 

Taking into account the learner’s characteristics by individually adapting learn- 
ing methods has a big influence on the learner’s performance (Issing 2002). The 
importance of the adaptation to the learner’s characteristics (also called 
personalisation) was shown in several studies. For example, the adaptive subject 
material combined with adaptive styles of presentation supports students to 
improve their learning achievements and increases learning efficiency (Tseng 
et al. 2008). Furthermore, the importance of adaption to individual learning prefer- 
ences of a learner regarding visualisation and verbalisation has been proven (Plass 
1998). Through a requirement analysis it has been found out that the learner’s 
knowledge, goals and tasks, language, interests, and learning styles are important 
factors of personalisation approaches (Hover and Steiner 2009). 

Supporting SRL in the right way is a crucial factor. On the one hand, it means 
providing enough freedom for the learner, in order to stimulate motivation. How- 
ever, on the other hand, too much freedom may be overwhelming and an appro- 
priate guidance or even adaptation is usually needed to make the learning process 
effective and efficient. The concept of guidance and freedom is important because it 
has been recognised that highly motivated learners attain a better learning perfor- 
mance if they have more control over their learning and are more autonomous 
(Issing 2002). On the other hand, some learners show difficulties in carrying out 
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concrete meta-cognitive activities, such as planning, goal setting, monitoring, 
evaluating, and as a result often perform less successfully than would be anticipated 
(Bannert 2006). Such learners are in need of guidance when learning. Furthermore, 
less motivated learners can also attain an improved learning performance if they 
receive more guidance. Keeping these reported findings in mind, the individual 
support for learners should be tailored to suitable degrees of guidance and freedom. 
In this respect, the learner should be offered an optimal and balanced level of 
control and autonomy for their own learning process. 

Motivation is a highly relevant aspect for achieving good learning outcomes and 
for performing SRL activities. Winne and Hadwin (2008) showed the positive 
impact of motivation on student’s attention to their learning progress, on the 
progress itself, and on the experience of satisfaction and positive affect. For the 
use of SRL activities, a learner has to be motivated as these activities require 
additional time and effort. Ryan and Deci (2000) describe intrinsic motivation as 
one of the most important aspects regarding learning because it is the prototypical 
manifestation of the human tendency towards learning and creativity. Behaviours 
of intrinsically motivated learners are freely applied without the necessity of 
separable consequences. For intrinsic motivation to develop, there is need for 
autonomy, competence, and relatedness. However, there is also a need for extrinsic 
motivation and especially a good balance between extrinsic and intrinsic moti- 
vation (Covington 2000) 

Another important factor for SRL is collaboration. According to Dillenbourg 
(1999), collaborative learning comprises individually performed activities and also 
extra activities that are generated by interaction among peers. These collaborative 
activities trigger additional cognitive mechanisms, which may appear more fre- 
quently in collaborative learning situations than in individual learning. Students 
working in cooperative learning situations compared to individualistic or compe- 
titive learning situations have a higher performance at the mastery and retention of 
material, are more often using focusing, elaboration, and meta-cognition strategies, 
and develop ideas or solutions which are not gained if they work on their own 
(McConnell 2000). Collaboration can also create both intrinsic and extrinsic moti- 
vation and is an essential strategy for stimulating curiosity, emulation, attention, 
persistence, opening new perspectives, and increasing self-efficacy (Waite and 
Davis 2006). 

All these aspects are relevant for the psycho-pedagogical approach of ROLE. 
The process model and related cognitive and meta-cognitive learning activities 
have been explicitly modelled. The concept of guidance and freedom as well as 
adaptation is achieved through recommendations of activities and resources. Other 
concepts, such as motivation and collaboration, are implicitly targeted through the 
use of technology. 
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Related Work and Technology 


This section gives a short overview on technology-enhanced learning (TEL) solu- 
tions discusses them from the SRL perspective. Technology plays an important role 
for SRL because it has been reported that TEL environments can provide opportu- 
nities to enhance SRL competences for students, especially meta-cognitive com- 
petences (Bannert 2006). 

Since the early 1970s, researchers have adopted the educational model of human 
tutors and started to implement it in intelligent computer-based instructions 
(Corbett et al. 1997). The goal of those Intelligent Tutoring Systems (ITS) was to 
engage students in learning activities and to interact according to their behaviour. 
ITSs were supposed to bring intelligence to computer-based instruction, especially 
in the knowledge of the subject domain as well as the tutoring principles and 
methods of their application (Anderson 1988). This led to the development of 
four basic components: the domain model, the student model, the tutoring model, 
and the user interface model. An important research strand is also the work on 
adaptive hypermedia and adaptive Web and how it is applied in the educational 
context (Brusilovsky et al. 2007). Strategies and systems (e.g. as described in Albert 
and Schrepp (1999) and DeBra et al. (2006)) have been developed that aim to adapt 
its content and behaviour to the individual student. In the light of SRL, these 
solutions do not support the learner. They do not allow freedom or autonomy, but 
guide the learner through the learning process. They also do not give any hint or 
explanation regarding SRL. 

A different development of TEL solutions is Learning Management Systems 
(LMS). They primarily focus on distributing learning content, organising the 
learning processes, and serving as interface between learner and teacher. In educa- 
tional institutions LMSs have become very popular and are used in many univer- 
sities and schools (Paulsen 2003). Examples of LMSs are Moodle, CLIX, 
Blackboard, WebCT, Sakai, ILIAS, and LRN. They all have in common that 
different tools are integrated in a single system, such as discussion forums, file 
sharing, whiteboards, chat, and e-portfolios (Dalsgaard 2006). These tools together 
with learning content are bundled by teachers or tutors, leading to a centralised and 
standardised learning experience (Guo et al. 2010). In contrast to ITSs or adaptive 
systems, no automatic guidance is provided by LMS. The guidance the learners get 
is provided by teachers or instructors who prepare courses. On the other hand, 
learners have more freedom and autonomy through some tools integrated in LMSs. 
For example, collaboration tools allow to discuss certain topics. 

In contrast to an LMS, personal learning environments (PLEs) strive for a more 
natural and learner-centric approach. PLEs are online systems that combine ser- 
vices as well as tools that enable users to create their own customised learning 
system. The teacher is no longer the provider of knowledge, but rather a mediator 
between knowledge and students. On the other side, the student is responsible for 
organising information and their own learning. In recent years, attempts have been 
made to build PLEs based on mashup designs. An example based on social media 
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tools is eMUSE (Popescu and Cioiu 2011), which integrates Web 2.0 tools into a 
single system. It claims that such tool integration leads to a sense of community and 
thus increases success and retention rates. Furthermore, eMUSE offers support for 
self-monitoring and self-evaluation by providing feedback on learning tasks, which 
is supposed to increase learning success and motivation. A further example is the 
PLE developed at the Graz University of Technology (Ebner and Taraghi 2010). 
This PLE allows for selecting widgets from a repository and adding them to a 
personal space. Besides some general purpose widgets (similar to the tools in an 
LMS), domain-specific widgets have been created by students in university courses. 
It also allows for logging and analysing students’ activities performed on these 
PLEs (Softic et al. 2013). Users can manage their PLE and, therefore, take greater 
control of their own learning approach. 

Specific technology that aimed supporting SRL was developed in several 
research projects. For example, the iClass project aimed at developing an intelli- 
gent, cognitive-based open learning environment that supports the planning, moni- 
toring, and reflection processes of a learner and at the same time personalise the 
learning process to the respective learners’ preferences and needs (Aviram 
et al. 2008). In order to achieve personalised recommendations and reflection 
support, a competence model has been used that is basis for individual guidance 
(Steiner et al. 2009). This competence model is based on Competence-based 
Knowledge Space Theory (CbKST) (Heller et al. 2006) that structures competences 
through prerequisite relations. Visualisation tools have been created that display the 
competence structures and let the learner select learning goals and learning paths 
(Nussbaumer et al. 2008). 

The ROLE approach differentiates from these solutions in the way that it pro- 
vides a maximum of technical flexibility and provides personalised learning support 
at the same time. A psycho-pedagogical model is integrated with the technology, 
which leads to range of freedom and guidance to be chosen by the learner. 


Models and Framework 


Self-Regulated Learning Process Model 


A process model (Fruhmann et al. 2010) has been developed that takes into account 
requirements of the ROLE approach to create and use responsive open learning 
environments that support SRL. The SRL process model describes learning as a 
self-regulated process consisting of four learning phases: (1) defining or revising 
the learner’s profile information and setting up an initial individual learning plan, 
(2) finding and selecting learning resources, (3) working on selected learning 
resources, and (4) reflecting on the applied strategies and reached achievements. 
The model is depicted in Fig. 1. This approach follows the cyclic SRL model 
proposed by Zimmerman (2002) and was slightly extended and modified. First, the 
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Fig. 1 SRL process model. This diagram depicts the SRL process model and related strategies 


phase when learners create their own learning environment is considered as a 
separate phase because this is a central part of ROLE. In the Zimmerman model 
this phase would be related to the forethought phase. Second, the personalisation 
aspect has been included by defining a user model consisting of models for 
describing competences, learning activities, and goals. 

In addition to these phases, a taxonomy of learning strategies, learning tech- 
niques, and activities has been defined, in order to operationalise the four phases. 
The phases are associated with learning strategies (see Fig. 1) and each strategy is 
related to learning techniques and learning activities. We consider learning acti- 
vities as the applications of learning techniques. This approach is based on the work 
of Dabbagh and Kitsantas (2004) on SRL processes and Mandl and Friedrich (2006) 
on SRL strategies and techniques. These findings have been adapted and integrated, 
in order to serve as a comprehensive and integrated model for technology-support. 

Using this theoretical background, we have defined nine SRL strategies and 
structured them in three groups, which are cognitive strategies, meta-cognitive 
strategies, and resource management (see Fig. 2). The group of cognitive strategies 
include organisation, elaboration, and rehearsal tasks of learning topics. The group 
of meta-cognitive strategies include goal setting, self-monitoring, and regulation 
tasks targeting the control of the own learning process. The group of resource 
management strategies include time management, help-seeking, and enabling 
(or environment preparation), which means that learners take care for their learning 
resources. These strategies are connected to SRL phases, which enrich the meaning 
of these phases with a clearer notion. For the instantiation of learning strategies and 
learning techniques, two complementary approaches have been used. 

For each of the nine listed learning strategies, a variable number of learning 
techniques are assigned. For instance, elaboration can be done with the following 
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Fig. 2 SRL Strategies. This diagram displays the SRL strategies and their relations to each other 
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Fig. 3 SRL techniques. This diagram gives an example of an SRL strategy and six associated 
learning techniques 


learning techniques: paraphrasing, creating analogies, producing questions, note- 
taking, brainstorming, and collaborative learning (see Fig. 3). A complete list of 
learning techniques and their assignment to the SRL strategies can be found in the 
ROLE ontology (see section “Learning Ontology”). 


Applying the SRL Process Model 


Using ROLE learning environment learners are interacting with a learning environ- 
ment that provides various resources like contacts to other actors, appropriate tools/ 
services, and suitable artefacts for the acquisition of knowledge, skills, and the 
fostering of competence development. For the personalisation and the preparation 
of a learning environment, learners have to set up their learning space in a 
meaningful. Especially in SRL situations, where learners take responsibility for 
the learning process and learning outcomes, the competent application of learning 
strategies is considered helpful for learners and can lead to better learning perfor- 
mance (Weinstein et al. 2005). 

In SRL situations the definition of the learning goal before the selection of 
learning tasks and the application of learning activities is very important. If learners 
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Technique 


Fig. 4 Applying SRL strategies and techniques. This figure explains how SRL strategies and 
learning techniques are applied in the learning process to achieve a learning goal 


are not aware of their learning goals and if they do not know how to reach them, 
learners incline to apply more or less habitual learning strategies that may not be 
appropriate for processing a learning task instead of adapting or making up new 
strategies. The SRL process model and the related learning strategies and tech- 
niques are used to guide the learning through the learning process (see Fig. 4). First 
learners set a goal. Then they choose from a set of cognitive, meta-cognitive, and 
resource management strategies. The strategies are related with respected learning 
techniques and learning tools. In this way a sequence of learning activities (strat- 
egies and techniques) is created that should lead to reach the learning goal. Through 
recommender systems (see section “Support Strategies”) the learner can get help 
during the selection process. 


Competence Model 


A competence model consisting of different kinds of competences takes into 
account domain knowledge, the ability to learn with tools, and SRL competences. 
The distinction of the competence types originates from the fact that learning 
happens on different levels. We distinguish between (1) domain knowledge and 
related competences, (2) the ability to learn with specific tools, and (3) the ability to 
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learn in a self-regulated way. Therefore, three types of competences are defined: 
domain competences, tool competences, and SRL competences. 

The domain competence describes the knowledge of a learner regarding a 
certain domain-specific topic. In detail a concept (e.g. from a concept map) is 
assigned with a level from the European Qualifications Framework (EQF).* The 
tool competence defines the ability how a learner can apply a learning technique 
with a certain tool functionality. Formally spoken, a tool competence is a pair of a 
related learning technique and an EQF level. For example, it can be defined how 
well a learner can debate with a tool that allows for video conferencing. The SRL 
competence defines the ability regarding one of the nine learning strategies 
described above. 


Learner Model 


In adaptive systems, user models are often designed as overlay models where user 
information relies on and is described by conceptual information (e.g. concept 
maps). Our learner model follows this scheme, but also relies on the learning 
taxonomy and its elements. This is necessary as the information about the learner 
is not only given at domain level, but also on the level of cognitive and meta- 
cognitive activities (the application of the learning strategies and techniques). The 
learner model consists of four elements. First, the competence state describes the 
available competences of the learner taking into account that there are three 
different types of competences. Thus, a learner is assessed regarding domain 
knowledge, the ability to learn with tools, and the ability to learn in a self-regulated 
manner. Second, the goals of a learner are also described with these competences, 
e.g. the learning goal is expressed in terms of the competences a learner wants to 
achieve. Third, the learning history is described by the learning activities (applied 
strategies and techniques) a learner has performed, the tools or widgets she has 
used, and the competences she has attained. In contrast to plain log data, this 
learning history provides more insight because it describes the history in terms of 
learning not just the particular interactions, but also regarding learning activities 
and used resources. Fourth, pedagogical parameters describe individual preferences 
and properties of the learner. Examples are information about preferred tools, 
learning groups, or guidance mechanisms. An overview is given in Fig. 5. 


3 http://ec.europa.eu/education/lifelong-learning-policy/eqf_en.htm 
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Fig. 5 Competence ontology. This diagram depicts the relevant elements of the competence 
model and how they are technically interrelated 


Learning Ontology 


The formal representation of the models listed above (competences, strategies, 
techniques, goals) is expressed as ontology in RDF/OWL* format and consists 
of the classes and properties. An overview of this ontology is depicted in Fig. 5. 
More details on the learning ontology can be found in Dahn et al. (2013). The 
Competence class is the superclass of the three types of competences. It allows 
specifying title and description via reusing the title and description properties from 
the Dublin Core® terms vocabulary. Furthermore, it provides two properties that 
allow for a generic definition of the competence. First, a topic property is used to 
relate a specific competence with a generic object (owl:Thing). Secondly, the 
proficiencyLevel property is used to express the extent to which a user has a specific 
competence. The generic Level class is subclassed by the EQF class to indicate the 
competence level. 

In order to relate a competence to a learner, a Person class is used that has 
the properties acquiredCompetence, goalCompetence, uses, applies, and 
hasParameter. 

In this way competences are defined that a learner has already acquired and 
competences that a learner should attain, the tools a learner has used, the learning 
techniques she has applied, and the personal preferences. The DomainCompetence 
class allows for defining a domain competence. It is a subclass of the Competence 
class and inherits title, description, topic, and EQF level specification. In this way 


E http://www.w3.org/RDF/ 
2 http://www.w3.org/TR/owl-overview/ 
6 http://dublincore.org/ 
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competence can be defined by using the topic property to assign the related domain 
concept and the proficiencyLevel property to define the EQF level. The 
ToolCompetence class allows for defining a tool competence. It is a subclass of 
the Competence class and inherits the title and description specification. Following 
the definition of tool competences, the topic property is used to relate tool func- 
tionalities and the technique property is used to relate a LearningTechnique. The 
SRLCompetence class allows for defining SRL competences. It is a subclass of the 
Competence class and inherits the title and description specification. Following the 
definition of SRL competences, the topic property is used to relate to SRL strategies 
and the inherited proficiencyLevel property is used to relate an EQF level. Since 
SRL competence can be related to different types of SRL strategies (cognitive, 
meta-cognitive, and resource management strategies), appropriate subclasses of the 
SRLStrategy class have been defined. 


Mashup Guidelines 


The mashup of a learning environment is an important feature. It can affect the way 
of learning not only by its pure use, but how it is compiled. It is assumed that 
compiling widgets to a learning environment or bundle relate to different psycho- 
pedagogical aspects and educational components. Taking into account these edu- 
cational components, some essential guidelines have been developed to provide the 
learner with recommendations for a psycho-pedagogically sound mashup (Berthold 
et al. 2012). These guidelines should be taken into account by the learner creating a 
bundle for herself and by teachers or tutors creating bundles for learners. It is also a 
basis for recommender technology that can apply these guidelines. 

Guideline 1: Cover all phases of the SRL process model: As described in section 
“Theoretical Background”, learning is more successful if different cognitive and 
meta-cognitive strategies are applied. Hence, a mashup design should contain 
widgets that support different learning strategies. For example, one widget for 
each phase of the SRL process model. This could be materialised in a setting 
with a widget for goal setting, a widget for content searching, a widget to learn 
how to find content, and a widget where the learning process is reflected. 

Guideline 2: Tool competence: Another educational component that comes into 
play is tool competence. Tool competences are abilities a learner has if she has 
knowledge about a tool, is able to perform a learning activity with a learning tool in 
a domain or domain-independent context, and is able to choose a tool or tool 
functionalities unassisted according to her learning or working task, e.g. the learner 
can use a tool for goal setting. If a tool competence is lacking for a certain widget, 
the learner will have problems to effectively use this widget. 

Guideline 3: Number of widgets: The number of widgets and their different 
functionalities might overtax the learners, especially if they are not intrinsically 
motivated to perform the learning task. Hence, only an appropriate number of 
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widgets should be added to a widget bundle. The range of the appropriate number of 
widgets within a widget bundle depends on the learner. 

Guideline 4: Domain aspects: In the third phase of the SRL process model, the 
actual learning process takes place. This phase is influenced by learning goals. 
Learning goals are defined in the first phase of the SRL process model. Such 
learning goals are mostly domain specific and refer to a special context. Thus, 
widgets in the bundle should be in line with the domain aspects of the predefined 
learning goals. 


Support Strategies 


In the previous section it has been described how SRL can be operationalised. In 
this section concrete approaches are reported how SRL is supported by using the 
models and framework described above. They establish the ground basis for the 
technology and concepts to support learners in a scientifically driven way and 
provide learners with according technology and material to guide them through 
the learning process. 

Learners who are able to understand the SRL process and the related learning 
activities and who are able to perform them on their own can navigate freely 
through their learning processes. However, this requires the availability of a high 
degree of SRL capabilities (availability of the respective SRL competences). Since 
it cannot be assumed that all learners already have these abilities, guidance mecha- 
nisms are needed. Such guidance is often needed especially when learning with 
technology-enhanced environments (Bannert 2006). 

According to the experiences made in the test beds and at several workshops (see 
section “Related Work and Technology”), a variety of guidance strategies is 
necessary. Both technical and human support is needed depending on the learner 
and the situation. Additionally training and introductory material turned out as 
useful to increase initial motivation for new ways of learning. 


Motivational Video 


This section explains an initiative to make learners acquainted with SRL. 
According to our experience (in test beds and SRL surveys), the concept of SRL 
is new to most lectures and learners. Lifelong learning, non-formal learning, etc. are 
buzz words society talks about, but SRL is apparently a term more common in 
science and pedagogical research. For this reason, SRL needs to be introduced and 
actually promoted to learners, teachers, and a broader community to point out 
benefits of such a learning approach. An introduction can be done in many 
different ways. 
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Fig. 6 SRL video. This figure is a screenshot of the video that introduces and explains SRL in the 
context of ROLE 


Teaser videos were used to explain the concept of SRL on a basic level for 
learners who are completely new to SRL. A video’ has been created that focuses on 
the explanation of SRL by comparing it to a city travel (see Fig. 6). Two people are 
examining a city and its sights. While the first one is attending a guided tour, the 
second one defines the goals and plans on his own and does a city walk without 
external support. This analogy is explaining both the concept of self-regulation and 
the concept of guidance. Further videos have been developed in the course of the 
ROLE project, in order to explain different aspects of PLEs and SRL*. Especially, 
tutorial videos have been created that explain how to use ROLE technology. 


Courses and Training Material 


An SRL course is another method to reach learners and provide them with assis- 
tance and knowledge about SRL and SRL tools in a compact way. The goals of such 
a course are to introduce the idea of SRL and enable them to build their own 
learning environment. The content of such a course can be a brief SRL explanation, 
an explanation of different learning models or learning strategies, and how they can 
be used with ROLE technology. 

At the Open University UK test bed such a course has been created as Online 
Course and as eBook. This course explains the basic concepts and also lets learners 


7 http://www.youtube.com/watch?v=jTal vOH6JjA 
8 https://www.youtube.com/user/ROLEProject 
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try out to create PLEs on their own. In this way learners can train SRL behaviour 
because they get step-by-step explanation and can try it out immediately. 


Preconfigured PLEs 


Predefined PLEs are already compiled bundles of widgets to fulfil a certain learning 
need. Therefore, they are usually assembled with widgets that cover a certain 
domain (e.g. history, chemistry, or language learning). Such bundles are typically 
created by teachers or peers. Teachers have the chance to prepare a bundle suitable 
for the learning topic. In this way, the guidance is based on the preparation of whole 
bundles that are suitable for individual learners. A special case is Layered PLEs that 
consists of a set of widget bundles. Each set may be dedicated to a certain learning 
strategy of an SRL process phase. 


Widget Store 


The ROLE Widget Store is a Web-based online catalogue that allows to manage 
and index widgets. It provides a user-friendly interface to a widget repository that 
simplifies the discovery of widgets. The functionality of the widget store includes 
listings of widgets, categorisations, searching by widgets or keywords, and compi- 
lation of widgets into bundles. Users can add widgets from the widget store to PLE 
systems. From a social media point of view, the store is also the place to collect and 
share user tags, comments, and ratings. A widget creator and developer can add a 
widget to the store by adding its reference (URL) and metadata. 

In order to provide guidance for learners in searching and selecting widgets for 
their PLEs, widgets can be tagged with metadata describing the purpose of the 
widgets. The first type of tags is a widget categorisation consisting of seven 
categories. The categories were derived from the SRL learning process model 
and are assigned to its phases. As described above a PLE should consist of widgets 
not only for one learning strategy, but widgets for different strategies should be 
included. The categorisation system is a useful way to follow this guideline because 
users get quick access to widgets for the specific purposes. They can browse the 
store and add widgets just by navigating to different categories. 

In addition to the widget categories, functionalities described in an ontology 
are used to represent features of widgets (e.g. text editing, video chat). These 
functionalities are derived from a survey of existing widgets and from an analysis 
of the ontology. The SRL techniques are related to functionalities so that the 
ontology and the Widget Store share the same set of functionalities. 

The third type of metadata is the domain concept describing widgets regarding a 
knowledge domain they are related to. Widgets can be either generic (e.g. text 
editor) or targeting specific learning domains (e.g. French language). As some 
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widgets can hardly be described by tool categories or functionalities alone, a 
categorisation based on learning domains is introduced. The service of DBpedia’ 
is used to allow users the tagging of widgets by semantically unique learning 
domains supporting them in search for specific tools. 

The user interface of the Widget Store allows for searching and filtering widgets 
according to categories, functionalities, and domain concepts. A list of widgets is 
listed according to the applied search. Additional filters can be applied regarding 
the metadata available for the listed widgets. The metadata for each widget is 
shown in the search result list, that is, category, functionalities, domain concepts, 
rating, title, and description. 


Mashup Recommender 


The Mashup Recommender widget (MR) (Fig. 7) can be seen as a filtering system 
that provides widgets that can be added to the PLE depending on the used template. 
The MR contains predefined templates, e.g. an SRL template. This template could 
include the four phases of the SRL process model. If the user selects such a phase, 
related widgets that support this phase are displayed. For example, if the learner 
selects the planning phase, calendar widgets, and To-Do-Widgets could be 
suggested by the MR. For this purpose, the MR uses SRL entities from the 
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Fig. 7 Mashup Recommender. This screenshot displays the Mashup Recommender widget in the 
middle and a widget that was recommended on the right side 


3 http://dbpedia.org/ 
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ontology. The ontology service is questioned for the respective functionalities of 
the SRL entities (learning strategies, techniques, and activities) and the widget store 
returns the associated widgets. Such templates can be created using a special 
authoring tool. 

The MR can be used to provide guidance on different levels and for different 
stakeholders. A high level of guidance is the preparation of complete predefined 
PLEs based on a specific template by a teacher or tutor. Later the tutor can share this 
PLE with her students who can use it or modify it. A lower level of guidance can be 
provided if the teacher just shares the template with the students, so that they have 
to create their own PLE. For example, a teacher could select the SRL entities goal 
setting, resource searching, note taking, and reflecting for a template. Teachers or 
learners using this template could easily search these SRL entities for widgets and 
include them in a PLE. In this way the PLE consists of widgets for each SRL entity. 
Learning strategies are on a higher abstraction level, which results in a greater 
number of widgets that can be recommended. Learning techniques are on a lower 
abstraction level, which leads to a smaller number of related widgets that can be 
recommended. While in the first case the learner gets more widgets recommended 
and thus less guidance, in the second case the level of guidance is higher because of 
the smaller number of recommended widgets. For a detailed description of the MR 
and its technical background, see Nussbaumer et al. (2014). 


Activity Recommender 


The Learning Activity Recommender guides the learner through the learning 
process by recommending learning activities related to the SRL process model. 
The learner is guided by means of a step-by-step approach of how to cope with a 
problem. In contrast to a direct instruction, the learner can decline to accept 
learning activities and can choose between alternatives and will not be penalised 
for varying his learning steps from what is suggested. The Activity Recommender 
(AR) consists of two widgets, the Activity Recommender widget and the To-Learn 
List widgets (Fig. 8). 

In contrast to collaborative and content-based filtering approaches handling large 
community-generated data sets, the Activity Recommender is working with data 
predefined and structured by the educational experts according to the educational 
approach described in the last section. These experts prepare the recommendations 
by defining learning strategies, techniques, and matching activities for learning 
tasks using an authoring tool. The Activity Recommender guides the learners 
through the learning process by recommending learning activities and assists 
them to compile a learning plan. In contrast to direct instructions, the learners 
have a free choice which recommended learning activities they want to perform. 
When a learner has decided to use a recommended learning strategy, the respective 
learning activities can be sent from the Activity Recommender to the To-Learn list 
widget. The To-Learn list widget allows learners to compile an individual learning 
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Fig. 8 Activity recommender. The To-Learn List widget and the Learning Activity Recom- 
mender widget are running in the ROLE SDK environment 


plan. The widget enables to add, rearrange, delete, and rename recommended 
learning activities or to add own activities, e.g. reminding them to take a break 
after a brainstorming session. A learning activity is described with a short summary 
and a longer descriptive text. Every learning activity has a status that is either not 
started, started, completed, or cancelled. Moreover, it is possible to specify the 
learning activities by adding sub-activities on the lower hierarchy levels. Some of 
the activities are highlighted in red colour, which means there are further 
recommendations available for this activity. Displaying these recommendations 
can be triggered by clicking on a highlighted entry. Finally, the entries of the 
learning plan can be sorted by status, date, or manually. 


SRL Text Reader Bundle 


The SRL Text Reader Bundle (see Fig. 9) is a predefined widget bundle that 
supports SRL by providing feedback on the SRL activities. The bundle captures 
certain SRL activities and displays them in a graphical way to make the user aware 
about the activities she performs. The main widget is Text Reader where learners 
can read and annotate texts. These texts and related concepts are defined in a 
domain model on a backend service. The Self-Evaluation widget allows for relating 
the assigned tags with concepts from the domain model and to determine the 
proficiency level for each concept. In this way, the learner evaluates herself 
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regarding her own domain competences. The search widget allows searching 
additional resources for the domain by searching related tags and concepts. All 
performed activities are recorded and stored in the user model. The visualisation 
widget follows an Open Learner Model approach and gives feedback to the learner 
about her learning process. In addition, the visualisation widget displays the texts 
that have been annotated and the concepts that have been used for self-evaluation. 
Guidance is provided by delivering a complete bundle of widgets that support the 
whole SRL process. 

From the user model perspective some different types of information is saved for 
further usage. The activities a learner performed are saved using the activity schema 
outlined in the ontology definition in section “Support Strategies”. The concepts 
coming from a domain model in the background and used for self-evaluation are 
stored together with the proficiency level as domain competence. The tags related to 
certain texts are saved as generic information. All this information is used for 
keeping the user data persistent and visualising the analysis in the Self-Reflection 
widget. 


SRL Monitor 


The SRL monitor provides support to develop self-awareness about the performed 
learning activities. The goal is not only to monitor and visualise the observable 
actions (as saved in log data), but also to monitor the cognitive and meta-cognitive 
activities that are not directly measurable. To this end, the measurable actions are 
mapped to cognitive and meta-cognitive learning activities from the ontology. To 
be precise, the key actions extracted from the log data analysis (based on an 
algorithm that clusters the log data) are mapped to elements of the learning 
ontology. The mapping is partially done by the learner herself, but also supported 
by an algorithm that takes into account the previous manual assignments. The goal 
is to make learners aware about their cognitive and meta-cognitive learning 
activities. 

The screenshot displayed in Fig. 10 shows two views of the SRL Monitor. In the 
first view, the SRL Monitor displays the learner’s captured log data in a sequence. 
Then the learner can select which learning technique she has actually applied. 
Based on these selections, reasoning is done regarding the applied learning strate- 
gies. Since there are only nine learning strategies, a comprehensive overview of the 
learner’s behaviour can be given. This overview is graphically shown in the second 
view of Fig. 10. In this way learners get feedback about their learning behaviour and 
might rethink their learning process if some learning strategies never appear on the 
graphical profile. 
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Fig. 10 SRL monitor. Screenshots of the two views of the SRL monitor widget are shown 


Evaluation Results, Challenges, and Barriers 


This section gives insight in the experiences made with the psycho-pedagogical 
model and the support strategies described above. It starts with an overview of 
evaluation studies of a few support strategies. Then it describes the experiences 
made in the test beds, in summer schools, and at conference workshops. More 
detailed evaluation results are described in the Chaps. 4—7 of this book. 


Selected Evaluation Results 


One evaluation study (Kroop 2013) focused on the usefulness of the developed 
Mashup Recommender (MR) widget (see section “Mashup Recommender”, Fig. 7) 
and the SRL template it provided. The study compared teachers’ and students’ 
acceptance for the MR and its underlying SRL strategy. Altogether the findings 
show a broad acceptance of the MR concept by both teachers and students. When a 
group of teachers and a group of students independently from each other tested a 
variety of PLE-scenarios, the MR was recognised to be the only tool which 
consistently supports the creation of a PLE in a self-regulated learning way. Most 
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scenarios for creating a PLE provide too much freedom (especially in selecting 
relevant learning tools from a broad variety of open educational resources), which 
causes students as well as teachers to be exhausted easily from (individually) 
evaluating and selecting learning tools. Consequently, the provided SRL guidance 
by the MR was perceived as a relief in creating a PLE. Although neither the 
teachers nor the students already felt familiar with the MR widget, teachers and 
students independently from each other appreciated the well-conceived idea to 
support SRL while creating a PLE. The discussions as well as the final ratings of 
the MR concept revealed that teachers were even more optimistic than students 
regarding the potential of the MR concept to improve and ease (self-regulated) 
learning. The findings of this study confirm the findings of the below described 
TPSRL study: Not only do students need guidance how to learn self-regulated but it 
is also important to provide support to the teachers and to train them how to 
(efficiently) introduce SRL to students. Another study (Dahn et al. 2013) concerns 
the Activity Recommender (AR) (see section “Activity Recommender”). The 
objective of this evaluation was to uncover strengths, weaknesses, and suggestions 
for improvement of the proposed approach and implementation. Therefore, useful- 
ness of the learning support, usability of the developed software, and general 
feedback were questioned. The evaluation has been conducted in two different 
settings: The first evaluation took place in at the University of Koblenz-Landau in 
Germany and the second evaluation took place at the Shanghai Jiao Tong Univer- 
sity in China. Summarising the results, there was a clear indication that the students 
appreciate guidance and help during the learning process. Some of the students 
liked the idea of getting recommendations for learning strategies and techniques for 
improving their learning. However, some students did not understand that recom- 
mendations are suggestions and not “must-do” instructions. Usability turned out as 
being the weakest point of AR. Without supervision and human support, many 
students had difficulties with the proper usage of the AR. Concluding these results, 
the approach of recommending learning activities is useful for and appreciated by 
students, whose SRL competences are not very low, but who do not apply SRL 
activities during their learning process. However, very weak self-regulated learners 
need sound introduction to SRL not only on pure technological level, but also in 
terms of human explanation, an introductory course, or videos. 


Results from the Conference Workshops 


The workshop series at the ICALT conferences in 2011 10 and 2012"! aimed at 
investigating SRL in personal learning environments. The target audience consisted 
of researchers, developers, and users of learning environments who were interested 


10 http://dbis.rwth-aachen.de/SRL-ROLE201 1/ 
i http://dbis.rwth-aachen.de/S-ROLE2012 
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in building individual learning environments. The accepted papers were presented 
at the workshop and published in the ICALT conference proceedings. The presen- 
tation session was followed by a group and plenary discussion. 

The interactive format of the workshops helped to identify barriers of SRL. 
According to the participants, these barriers are mainly in the traditional educa- 
tional system that does not cultivate the self-regulation competences early enough. 
This system designed for industrial purposes does not suit modern requirements 
anymore, as it kills motivation and creativity in learners. Individualisation of 
learning is a major challenge in education and rapid technological development 
brings new opportunities how to address it. A good SRL solution should be 
personalised and adaptive, providing a right balance between the learner’s freedom 
and guidance, in order to both motivate and support the learner. This threshold is 
individual and context dependent. Therefore a spectrum of facilities is needed for 
various levels of SRL at different levels of education and in different contexts. 
Another key challenge is to prepare learners for lifelong learning when the teacher 
may not be available. They have to learn meta-cognitive skills, which are highly 
important. In this process, suitable scaffolding has to be taken into account. 

The PALE workshops at the UMAP conferences in 2011 and 2012 provided 
opportunities to identify and discuss various issues related to topics like pedagogic 
conversational agents, responsive open learning environments, and learner 
modelling. 

At PALE 201117 (Perez-Marin et al. 2011) the ROLE-related discussions 
focused on the usefulness of PLEs in order to support SRL. A big challenge is to 
find an appropriate threshold between the learner control and tutor guidance. 
Personalised support can be provided by means of design templates and recom- 
mendations. In any case, this support has to be effective but not intrusive, consid- 
ering the learner preferences. Moreover, these preferences are not static, but can 
change dynamically according to the context. Participants suggested focusing the 
research on two scenarios where learners are supposed to have certain capabilities 
to guide themselves: higher education and lifelong learning. It was also suggested 
to consider meta-cognitive competences as well as results of behavioural and 
cognitive psychology when designing learning environments. 

To summarise the outcomes from PALE 2012!° (Herder et al. 2012), a lot of data 
can be collected in the educational process, but we need to find ways how to use it 
reasonably and to develop useful services that make the learning process more 
effective and efficient, e.g. by predicting student outcomes in order to intervene. 
Novel personalised services and environments are needed especially in lifelong and 
workplace educational settings, in order to support informal, self-regulated, mobile, 
and contextualised learning scenarios. A big challenge is also adaptation consider- 
ing both long-term objectives and short-term dynamically changing preferences of 
learners. Here, open and inspectable learner models play an important role, 


12 http://adenu.ia.uned.es/workshops/pale2011/ 
1 http://adenu.ia.uned.es/workshops/pale2012/ 
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considering also learner motivation and affective state. In the case of pedagogic 
conversational agents, personalisation is fostered by the use of dialogues adapted to 
the specific needs and level of knowledge of each student. Here and in mentorship 
systems trust and reputation play a crucial role. 

At AHA 2012'* conference in Vienna a ROLE workshop"? on PLEs and the 
support of SRL took place. Workshop participants were mainly teachers and pro- 
fessors from Germany and Austria but also school consultants and technical support 
people from higher education institutions. At this workshop the participants had the 
opportunity to test and discuss a variety of ROLE tools for creating a PLE including 
tools for supporting SRL when creating a PLE. This workshop-session as well as an 
additional cross-validation workshop with students including teacher candidates 
from the University of Vienna revealed and confirmed success criteria as well as 
limitations and barriers for the uptake of the provided ROLE tools. The workshop 
results which explicitly included the perspectives and backgrounds of German- 
speaking countries and cultures were also presented and discussed with profes- 
sionals from Guatemala at a Workshop on Cloud Education Environments 
(WCLOUD 2012) in Antigua, Guatemala (Kroop et al. 2012). Interestingly the 
experts from Guatemala shared the perspective of their colleagues from German- 
speaking countries, especially the need for SRL-supporting technology. 

The essential success criteria for the uptake of the PLE tools were seen in the 
underlying pedagogical learning model: While participants were working with the 
provided tools and felt sometimes overwhelmed by the challenge to create a 
reasonable PLE, it became clear that a consistent model on SRL strategies and 
techniques as described in section “Models and Framework” was most needed. The 
idea to connect different stages of SRL (Planning, Searching, Learning, Reflecting) 
with corresponding ROLE tools for learning was seen most useful and most 
effective by almost all participants. A positive impact on learning by using 
SRL-supported PLE tools was especially seen in the following learning activities: 


¢ Getting started with a learning task in a meaningful way. 

¢ Keeping track of the own learning progress by following the provided learning 
strategy. 

« Improved time management and reflection. Limitations for the uptake of the 
provided ROLE tools as well as doubts on the learning improvement by using 
these SRL-supported PLE tools were assumed as well and mainly addressed the 
personality, attitude, and motivation of a learner: 


— Motivated learners will benefit from the provided PLE tools; it will improve 
their efficiency and outcome of learning. 


n http://ahakonferenz.at/fruhere-aha/aha-2012/ 


15 Sylvana Kroop, Marcel Berthold: Personalisierte Lernumgebungen. Unterstutzung von selb- 
streguliertem Lernen. 
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— Very motivated learners will not need the technology-based SRL-support 
(e.g. the Recommender Widget, see section “Mashup Recommender’, 
Fig. 7). 

— Less motivated or weak learners will not benefit from the provided PLE tools. 


Unfortunately the last assumption (about weak learners) was not well reasoned 
by the experts. But from the discussion context, it can be assumed that an intrinsic 
motivation of a learner is seen as a prerequisite to be successful when learning with 
PLEs. The idea that SRL-guidance provided by the PLE technology could trigger 
the motivation which is missed by weak learners was received with reservation. 

Crucial barriers especially teachers raised for the uptake of the provided ROLE 
tools were belonging to time-consuming concerns for (learning) activities such as: 


¢ To get in general used to the new PLE technology. 

¢ To create a useful PLE in order to use it for the content taught in school and for 
the most ambitious request. 

e To create and provide own SRL templates (see section “Mashup Recom- 
mender’’) which are adapted to a specific course or specific learning content. 


Time-related issues in general can be seen as typical criteria for the uptake of any 
new technology. However, interesting is the fact that the participating teachers were 
aware of the possible time-wise burden and the additional effort compared to their 
used traditional learning and teaching but still accepted the uptake of the provided 
ROLE tools. They explicitly recommended the SRL-supporting PLE tools. More- 
over, at the end of the workshop the involved teachers strongly expressed their wish 
to try out a PLE in their daily activities and thus created a mailing list in order to be 
informed and provided by further material, tutorials, online courses, or upcoming 
workshops on this topic. Altogether the experiences and findings from workshops 
gave important directions for the request and need of the research and development 
of innovative learning technology. 


Lessons Learned from the Test Beds 


Within the ROLE project, an instrument for understanding the perceptions of 
educators about SRL was devised. A questionnaire called “Teachers’ Perception 
of Self Regulated Learning” (TPSRL'®) was formulated, containing questions 
about how teachers perceive the SRL capabilities of their students, as well as 
about the challenges associated with teaching students with varied SRL compe- 
tences. This questionnaire was circulated among the teachers of the Higher Educa- 
tion ROLE test beds. The TPSRL survey set out to explore which factors potentially 
influence teachers’ assessments of their students SRL competence, how they see the 


16 http://fit-bsew .fit.fraunhofer.de/pub/bsew.cgi/39770946 
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relationship between students’ SRL competence and performance as well as which 
type of students in terms of SRL level the teachers prefer to teach. 

According to the results of the TPSRL survey, the majority of the teachers that 
responded had an awareness of SRL and independent learning. Most of them also 
recognised the important function of SRL alongside its significance to them as well 
as to students. Some respondents related SRL to increased maturity and, therefore, 
an acceptance of responsibility for one’s own learning. Several teachers regarded 
SRL as a joint venture, ie. learning together with students and “discovery 
together”. SRL was also seen to influence students’ ability to learn faster. Various 
teachers recognised that SRL could improve their students’ reasoning or 
questioning abilities as well as their concentration power and, therefore, their 
capacity to learn. Some alluded to a teacher’s moral duty to guide/show “right” 
path using SRL techniques, thus adding to the all round development of students, 
i.e. implying that it was an implicit responsibility of all teachers. 

The strategies the respondents to the survey use in order to motivate and support 
students in becoming self-regulated are quite varied. One of these strategies con- 
sists of providing specific academic study skills facilities outside the classroom, 
available as face-to-face as well as via online support from the university, 
i.e. blended approach. Additionally, several teachers direct students to online 
and/or library-based “learning to learn” resources. Some teachers prefer to offer 
less help to their students, thus encouraging them to take more initiatives and learn 
for themselves. “Leading by example” is also a popular strategy: the teacher 
indicates or offers different approaches to resolving subject-based problems but 
leaves the students to choose their own learning path. Finally, the majority of the 
respondents agreed that encouraging active learning through peer collaboration 
helps motivate SRL of their students. In particular, some teachers promote working 
together with their students, e.g. through semi-directed projects, or they encourage 
group work. 

The survey was also quite revealing with regard to some of the challenges in 
motivating higher education students to become self-regulated learners. Several 
respondents stated that many students are simply not equipped to learn at an HE 
level. They also mentioned that their students are reluctant to accept new methods 
of learning or change in their outlook on learning. On the other hand, most students 
expect to be provided with precisely defined learning materials and strategies by 
their teachers. One of the themes that emerged was that inspiring groups of students 
that have mixed learning skills is challenging in itself. Students with fewer SRL 
skills require more time to assimilate information or discover new methods of 
learning. This has implications for the teacher in terms of effort required to meet 
the needs of the entire spectrum of learning skills in the classroom. Most impor- 
tantly, the teacher may not have enough knowledge, experience, or personal 
confidence to include SRL in the delivered curriculum time frame. 
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Experiences in the Summer Schools 


In the years 2010-2012 several workshops on SRL were organised, mainly for Ph. 
D. students in the TEL area at the JTEL Summer Schools. Three key target groups 
were considered at these events, namely learners, teachers, and developers. At the 
workshops the objective was to demonstrate the conceptual and technical solutions 
for personalised support of SRL. In the introductory part we aimed to explain how 
psycho-pedagogical theories impact the design of learning environments, consid- 
ering principles of SRL. We emphasised the role of decision making for the quality 
of the learning outcome and that it should be supported properly to optimise 
learning for benefit of the learner. Specific support strategies for SRL were 
explained taking into account the various SRL concepts. The participants tried 
out the ROLE software aiming to create a PLE. Experience from these events has 
shown that Ph.D. students understood the ROLE approach quickly and managed to 
design their PLEs without bigger problems. 

In the practical part they elaborated approaches for personal support in small 
groups. The approaches were made with paper-and-pencil or with respective tools 
on their laptops. Each group presented their results followed by a discussion with 
the other participants. The group work was active and creative. However, it also 
turned out that it was really difficult for the students to find good and innovative 
solutions. As a conclusion it can be reported that the concept of personalised 
support for SRL can be explained to Ph.D. students, but it is very difficult that 
they find new solutions in a 3-h workshop. 

In 2010 we conducted a survey to test the impact of choice architecture on the 
responses of people and to find out what Ph.D. students think on some issues in 
TEL. Our respondents (advanced learners) mostly thought that too much freedom 
for the learner may be overwhelming and contra productive. Similarly, they agreed 
that learners needed pedagogical assistance. Finally, almost all of them appreciated 
availability of a competent tutor. In addition to these findings we could easily see 
how important it is to find good explanations of SRL concepts in order to achieve a 
common understanding. Slight changes in wording may dramatically change 
responses. At the same time this demonstrated that to identify real opinions and 
preferences of people may be a tricky issue and the concept of choice architecture 
can influence them essentially. 


Conclusion 


The ROLE project has aimed to achieve progress beyond the state of the art in user- 
centric responsive and open learning environments. A key objective of this inten- 
tion was the development of a psycho-pedagogical framework. The most important 
goal of this framework is to support learners to learn in a self-regulated way in 
responsive open learning environments. Based on this framework, a variety of 


Supporting Self-Regulated Learning 45 


support strategies have been developed. This approach is grounded on a literature 
review on SRL and experiences made in the test beds and in workshops with 
students, teachers, and researchers. 

One of the most important findings was that adopting the self-regulated way of 
learning can be very difficult especially for weak learners of learners not used to 
freedom in their learning process. In such situation technological support strategies 
alone are not sufficient and not successful. Therefore, a holistic support strategy is 
needed, which includes several factors: (1) Material (e.g. videos) that motivates 
learners and raises their attention for SRL. (2) Courses that provide step-to-step 
introduction to SRL, so that learners can practice and get use to a new form of 
learning. (3) Preconfigured environments that already support SRL, so that the 
learner does not have to start from scratch. (4) Peers should be included in the 
learning situation, which leads to communication and reflection on the own learning 
process. (5) Finally teachers (if available) should take care to support learners. It 
turned out that also teachers need introduction and awareness for SRL, because not 
all teachers do fully understand the concept of SRL. 

Five key aspects have been identified as essential for the psycho-pedagogical 
approach: Personalisation, degree of guidance, motivation, meta-cognition, and 
collaboration. Each support strategy takes into account at least one of these aspects. 
The SRL process model describes the learning process and thus serves as a 
backbone for support strategies. Based on these aspects, several support strategies 
have been developed (see section “Support Strategies”). However, these strategies 
are just examples demonstrating how support for SRL can be provided. Actually 
there is still room for new and more sophisticated support strategies. Future 
research and development can (and should) be done in this field. Taking into 
account the theoretical approaches described in this chapter, future possibilities 
for supporting learners will emerge. 

One of the limitations we faced during the project was the technology-based 
assessment of SRL competences. It turned out that automatically assessing SRL 
competences (e.g. by monitoring the learner when using the learning environment) 
is much more difficult than expected. Especially in this field there is room for 
further research. In order to overcome this problem, teachers were asked to estimate 
the SRL competences of their students and also students were asked to self-estimate 
their SRL competences. However, a method to automatically determine the SRL 
competences would still improve the learning process and would give new possi- 
bilities to support the development of SRL competences and to personalise the 
recommendations of learning resources and activities. 
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Abstract Evaluating highly dynamic and heterogeneous Personal Learning Envi- 
ronments (PLEs) is extremely challenging. Components of PLEs are selected and 
configured by individual users based on their personal preferences, needs, and 
goals. Moreover, the systems usually evolve over time based on contextual oppor- 
tunities and constraints. As such dynamic systems have no predefined configura- 
tions and user interfaces, traditional evaluation methods often fall short or are even 
inappropriate. Obviously, a host of factors influence the extent to which a PLE 
successfully supports a learner to achieve specific learning outcomes. We catego- 
rize such factors along four major dimensions: technological, organizational, 
psycho-pedagogical, and social. Each dimension is informed by relevant theoretical 
models (e.g., Information System Success Model, Community of Practice, self- 
regulated learning) and subsumes a set of metrics that can be assessed with a range 
of approaches. Among others, usability and user experience play an indispensable 
role in acceptance and diffusion of the innovative technologies exemplified by 
PLEs. Traditional quantitative and qualitative methods such as questionnaire and 
interview should be deployed alongside emergent ones such as learning analytics 
(e.g., context-aware metadata) and narrative-based methods. Crucial for maximal 
validity of the evaluation is the triangulation of empirical findings with multi- 
perspective (end-users, developers, and researchers), mixed-method (qualitative, 
quantitative) data sources. The framework utilizes a cyclic process to integrate 
findings across cases with a cross-case analysis in order to gain deeper insights into 
the intriguing questions of how and why PLEs work. 
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Introduction 


Among others, a critical success factor in technology-enhanced learning is the 
personalization of learning experience. As emphatically pointed out in the Leu- 
ven/Louvain-la-Neuve Communiqué of the Bologna Process 2020, “student-cen- 
tered learning requires empowering individual learners, new approaches to teaching 
and learning, effective support and guidance structures and a curriculum focused 
more clearly on the learner” (p.3). Personalization is also a key issue for 
implementing mechanisms to foster and increase activities in informal and lifelong 
learning networks. This implies a need for new technology-enhanced learning 
models that start from the learners and satisfy their unique needs in order to achieve 
a personalized learning experience for everyone. 

Recent discussions about technologies for learning have shifted from institution- 
managed learning management systems (LMS) to user-controlled social software 
for learning. Indeed, the advent of Web 2.0 technologies has phenomenally 
transformed the way in which users consume, communicate, collaborate, and create 
information and knowledge on the Web. These technologies have underpinned the 
emergent notion of Personal Learning Environment (PLE), which is characterized 
by qualities such as personalization, openness, responsiveness, flexibility, share- 
ability, interactivity, and sociability. PLEs can be perceived as both a technology 
and a pedagogical approach (Attwell 2007; van Harmelen 2006; Johnson 
et al. 2011; Johnson and Liber 2008; Schaffert and Kalz 2009) that aim to empower 
students to be in charge of their own learning by selecting tools and resources to 
create, organize, and package learning content, thereby meeting their personal 
needs and goals (McLoughlin and Lee 2010). 

Nonetheless, the high hope held for the PLE to be a key enabler for lifelong 
learning is yet to be shown because the research and practice on PLE is still 
evolving. Specifically, substantive claims about the power of PLE should be 
grounded in relevant case studies, which, however, are limited in number and 
scope (Johnson et al. 2011). The paucity of case studies and missing evidence on 
the success or usefulness of PLEs can be attributed to the lack of a comprehensive 
evaluation framework for PLEs. The difficulties of evaluating PLEs have been 
documented (e.g., Gillet et al. 2011; Giovannella 2011). While technical 
implementations have demonstrated some significant progress (see Chaps. 5 and 
8 in this volume), the empirical evaluation of PLEs lacks behind. Indeed, the 
development of an evaluation framework for PLEs poses several major challenges: 


— PLEs are not a stable technology that can be prepared and used in a controlled 
environment. In fact, PLEs do change over time and can be highly dynamic. 

— PLEs integrate other technological artifacts that are designed independently 
from each other and can stem from different providers. This leads to possible 
(unintended) interdependencies, usability issues, and update state problems. 

— PLEs are used to combine formal and non-formal learning contexts. Therefore 
the purpose of using a PLE can be highly heterogeneous, rendering systematic 
comparisons across different learners very difficult. 
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Fig. 1 Four perspectives (the TOPS model) for PLE evaluation 


To tackle these challenges, mixed-method and multi-perspective evaluation 
approaches are deemed relevant to address the complexity of PLE usage and its 
effects on learning behaviors and learning outcomes. 

Four main perspectives can be identified: technological, organizational, psycho- 
pedagogical, and social (short: “TOPS”), with each being informed by specific 
concepts and theories and subsuming certain methods and tools (see Fig. 1). They 
are elaborated in the following with reference to related work of the ROLE project 
(http://www.role-project.eu/). 


The “TOPS” Model for Evaluating PLEs 


In this section we delineate the individual perspectives of the TOPS model—with 
specific emphasis on their respective underlying conceptual and theoretical 
frameworks. 


Technological Perspective 


The technological perspective comprises two main aspects: utility and usability and 
user experience. It is to emphasize that the user-centered design (UCD) approaches 
underpin the work of PLEs, so not only end-users’ but also developers’ perspectives 
should be taken into account. 
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Utility 


Two major elements of utility to be evaluated are software and documentation, 
which are discussed in detail in the following. 

Software evaluation pertains to the functionality of different software compo- 
nents constituting PLEs, including widgets, widget containers, the widget store, 
libraries, services, tools, and the overall interoperability framework. It is essential 
to evaluate how useful these components are to enable end-users to accomplish 
specific tasks and goals. 

As already indicated above, the strict separation of end-users from developers 
can be seen as artificial (at least under the UCD approach), thus requiring an 
evaluation approach to look also at developers and “power” users who engage in 
customization, configuration, or even end-user-driven development (keyword: 
mash-ups). Typically, such developers and power users use configuration options, 
authoring tools, and APIs allowing for the mash-up of components to customize or 
even create new software artifacts. 

Specifically, we highlight a list of factors critical for the technical evaluation of 
software, which are adopted from constructs of the Information System Success 
Model (ISSM) (DeLone and McLean 2003) and the Technology Acceptance Model 
(TAM) (Davis 1989; Venkatesh et al. 2003; Venkatesh and Bala 2008) (see Table 1). 


Table 1 Constructs relevant to the utility evaluation of PLEs 
Construct Factor 


System quality Integration 
Portability 
Availability 
Performance 
Reliability 
Usefulness of features 


Completeness of features 


Security 


Information quality Usefulness of information 


Completeness 


Correctness 
Appropriateness of presentation format 


Service quality Responsiveness 
Support 
Feedback mechanism 
Trustworthiness 

Use Users, communities 


Functionality accessed 


Information items accessed 


Duration of use 


Frequency of use 
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An increasingly popular data collection method is automated monitoring: mon- 
itoring and data logging for capturing how frequently a service or a feature has been 
used and how often different significant events have occurred (Helms et al. 2000). 
More specifically, raw data of system use are recorded and then aggregated for 
computing measures for individual factors. For example, “mean time to failure” is a 
measure for the factor “reliability” under the construct of “system quality” that can 
be derived from monitoring data, including information about what and when errors 
occur. 

Contextual information gathering (i.e., information about the current situation 
where a learner deploys specific software) is also important. Noteworthy is that 
context has a technical and social aspect: which software/browser is used for 
accessing a PLE, which types of data are accessed, and which people interact 
with each other in a certain community. In principle, physical context information 
can also be recorded if the sensor data required would be available (e.g., GPS 
coordinates for spatial location). An example of including contextual data in 
evaluation is a factor “Browser Compatibility,” where the number of errors 
occurred related to a particular browser can be measured. Similarly, a factor 
“Widget Container Interoperability” can be measured by associating errors with a 
particular widget container where they occur. 

Where it is possible and does not infringe privacy and security regulations, it 
may often be safer to capture a broad standard range of monitoring data, especially 
as capturing technologies typically require no further human effort beyond initial 
setup and the setup is often already integrated into the related software. Subse- 
quently, such data can be selected, refined, and processed based on the actual needs 
and goals of an evaluation project. 

Croll and Power (2009) provide an elaborate list of metrics that can be used for 
monitoring usage of web-based technologies. Some of the key metrics relevant to 
PLEs are: user-generated content, content popularity, loyalty, search effectiveness, 
reflection, enrolment, conversion, and abandonment. 

There are a number of ways for collecting data for these metrics. Google 
Analytics are a free service to generate a comprehensive range of usage statistics 
for any web-based application. Following the insertion of a small JavaScript code 
snippet into a given web application, Google starts to record usage statistics 
(including simple demographic features and events). Some of the key aspects that 
Google can currently track are 


— Visitor Tracking: Demographics, conversion, uniqueness, loyalty, etc. 

— User Profile: browser, OS, screen resolution, Java availability, flash availability, 
connection speed, etc. 

— Events: frequency of use of specific event categories, events per visit, total 
number of events. 


One of the drawbacks of using analytics is the limited capability to provide data 
describing how users interact with content and tools (known as attention metadata) 
within their environments. Collecting contextualized attention metadata (CAM) 
will enable us to infer the ways learners use technologies and tools for specific 
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purposes. The CAM approach proposed by Wolpers et al. (2007) supports such 
tracking of attention metadata. This approach helps observe the user at the appli- 
cation level, enabling association of tool usage with content-specific behavior in 
context. The challenge of collecting observation data of user attention unobtru- 
sively can be resolved by the CAM approach through integrating the data-capturing 
process into a user’s daily working environment. This approach allows integrating 
data from web applications (e.g., by mapping the Apache open log file format to 
CAM) as well as from desktop applications. CAM helps track learning content 
usage, analyze behavioral patterns, provide similarity measures between users, and 
allow inferences about user goals. CAM data can be utilized to measure the 
effectiveness of PLE technologies in providing the learner with a highly responsive 
and personalized learning environment. CAM data can also be used to track and 
infer self-regulatory activities for measuring the effectiveness of the psycho- 
pedagogical model (Scheffel et al. 2010a, b). 

All measures that cannot be derived with automatic monitoring need to be 
obtained from users explicitly. The challenge is to identify appropriate techniques 
for survey data acquisition with the possible lowest obtrusiveness and highest 
intuitiveness for users. 

For instance, a lightweight “Requirements Bazaar” approach is integrated into 
the ROLE Widget Store (http://role-widgetstore.eu/) similar to other well-accepted 
systems such as Google’s Android Market or the Chrome Extensions marketplace. 
This is a valuable source of data since their users provide feedback on the quality of 
tools, services, and widgets using means such as rating scales, and—where appro- 
priate—free text comment boxes. 

Documentation evaluation looks into the availability and quality of technical 
documentation—a prerequisite for software to be accepted by end-users as well as 
developers. To encourage developers to contribute new learning technologies by 
mashing up existing software components, it is necessary to ensure that documen- 
tation is correct, complete, and tailored to developers’ needs. 

With regard to the development of web-based software components, developer 
documentation of the infrastructure usually includes the following items: 


— The set of initial documents (e.g., an overview of the underlying principles and 
overarching architecture). 

— The reference documentation with complete information on all supported fea- 
tures, usually in the form of API documentation. 

— The set of tutorials demonstrating how to use the technology for developments 
on simple and useful examples. 


Specifically, technical documentation should be tested by inviting developers to 
practical sessions, where they are asked to use the infrastructure and accompanying 
documentation to realize a small but motivating use case beyond basic tutorial 
contents. In such sessions, the developers who authored the documentation can 
serve as tutors to be consulted to discuss any problems arising. Such discussions can 
be used as individual interviews or focus groups to collect feedback on the quality 
of the software as well as documentation. This approach, however, does not scale to 
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large groups of developers. This is where the required alternative means such as 
online tools are preferred over presence workshops. 

Documentation of web-based software is usually supplemented by different 
technical means for communicating with the core developers of the original tech- 
nology, authors of the documentation (who often are also its developers), and 
developers deploying these software artifacts. For instance, developers use online 
forums to get in contact with other developers to report problems and ask for help. 
Besides bug reports, such comments often contain practical questions about how to 
accomplish certain tasks, thus indicating where the existing documentation could 
be unclear or incomplete. 

Further means to assess the utility of documentation is to directly integrate 
ratings, for instance, in the form of 5-star scales, like/dislike buttons or commenting 
functions, into the online documentation. In this manner, different factors from the 
dimension Information Quality can be surveyed. These (and additional) features are 
often already provided by software project management systems such as 
SourceForge, GitHub, and the like. 


Usability and User Experience 


First of all, it is deemed imperative to demarcate usability from user experience 
(UX)—two key concepts in the field of human-computer interaction (HCI). One 
main distinction is that usability targets instrumental quality, emphasizing the 
effectiveness and efficiency of task and goal attainment with interactive technolo- 
gies, whereas user experience targets non-instrumental quality (e.g., aesthetics), 
going beyond the traditional task-oriented focuses to address users’ affective and 
emotional responses (e.g., fun, pleasure, surprise, sad, happy) to interactive tech- 
nologies (e.g., Hassenzahl 2013). Hassenzahl’s (2005) oft-cited model on the 
pragmatic and hedonic quality illustrates similar arguments. Despite its decade- 
long history, some basic conceptual issues in UX are yet to be resolved (Law 
et al. 2009; Law, van Schaik & Roto, 2014). While a deeper exploration of such 
issues is beyond the scope of this chapter, here we highlight metrics and approaches 
relevant to the evaluation of PLEs. 

Noteworthy is that usability and user experience evaluations focus on the 
interaction design of technological components underpinning PLEs, which none- 
theless contribute to the holistic educative experience with PLEs (see also section 
“Psycho-pedagogical Aspect’). 


Usability 


The usability of different technological components of PLEs (section “Utility”) is 
to be evaluated based on a combination of metrics identified from the literature 
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(e.g., Nielsen 1994) and standards (ISO/IEC 25010:2011'; ISO/IEC 9241-1107: 
2006 and ISO/IEC 9241-210: 2010°). The metrics are listed as follows: 


— Learnability: The ability of the technology to enable users to learn with great 
ease how to assemble a PLE themselves. If users find it difficult to assemble a 
PLE, then the acceptance and uptake may be drastically hindered. Hence, the 
assembly process for such an open learning environment should be relatively 
straightforward for end-users. Some factors that enable us to ascertain 
learnability are consistency of user interface design and predictable system 
behavior. Learnability of PLEs is equally important for developers as for 
end-users. If developers find it difficult to use PLE software, they may not be 
able to create new widgets. 

— Efficiency: The ability of the technology to support users to be highly productive. 
Features such as consistent look and feel, consistent navigation, frequent feed- 
back, and availability of templates to help them quickly assemble their environ- 
ments can contribute to the overall efficiency of the PLE software. 

— Memorability: The ability of the technology not to require users to reinvest time 
in remembering how to use it after a period of nonuse. Closely related with 
learnability, memorability can influence the uptake and usage of PLE. The key 
success factor for PLE is to make the assembly process of the environment 
highly intuitive, using relevant standardized visual cues. 

— Error Tolerance: The ability of the technology to avoid catastrophic errors by 
making users reconfirm critical actions (e.g., deleting a software component) and 
to recover from errors by providing the “un-do” feature that allows users to 
reverse their actions. 

— Effectiveness: The ability of the technology to help users achieve their goals. 
Using PLEs, if learners are able to assemble and personalize their environments 
with ease, while at the same time they find the recommendations and rated/ 
ranked content useful for fulfilling their goal, then we can infer that the tech- 
nology is effective and that learners are likely to feel satisfied. More explicit 
methods are mentioned above in section “Utility.” 

— Flexibility: The ability of the technology to offer a range of services so as to be 
able to adapt to task changes. The ability of learners to seamlessly integrate and 
use a range of web-based tools and services for assembling their learning 
environments and for exporting/importing data as well as settings to other 
similar technologies. 

— Operability: The ability of the platform to allow users to operate and control it. 

— Satisfaction: The ability of the platform to be deployed by users without dis- 
comfort. It is highly subjective as compared with the other qualities listed above, 
which when realized to a sufficiently large extent, can contribute to overall user 


! Systems and software engineering: Systems and software Quality Requirements and Evaluation. 
? Ergonomics of human-system interaction: Part 110: Dialogue principles. 


° Ergonomics of human-system interaction: Part 210: Human-centered design for interactive 
systems. 
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satisfaction. Note that in addition to the system and service qualities, informa- 
tion quality can play a key part in user satisfaction, according to the ISSM 
(DeLone and McLean 2003). 


Usability evaluation methods comprise a range of usability inspection methods, 
user-based tests, and user surveys, which can be used to evaluate PLEs using the 
metrics described above. Inspection methods rely on experts, whereas user-based 
tests and user surveys, as the names suggest, involve end-users (an overview, see 
Holzinger 2005). 

Two commonly used inspection methods are heuristic evaluation and cognitive 
walkthrough. For heuristic evaluations, experts examine a system based on ten 
usability heuristics or principles that were originally derived from a large database 
of common problems. Violating any of such principles is identified as usability 
problem of which the severity is estimated so as to inform the urgency and necessity 
of its being fixed (Nielsen 1994). The major advantages of this method are that it 
can be applied throughout the whole development lifecycle and is, relatively, less 
time-consuming. In a cognitive walkthrough, experts analyze a system’s function- 
ality with a set of four questions (e.g., “Will the user notice that the correct action is 
available?”’) to estimate how the user would interact with the system (Lewis and 
Wharton 1997). A negative response to any of the questions suggests the identifi- 
cation of a usability problem. 

All inspection methods, as prediction methods, are prone to false alarms and 
results thereof are typically to be verified with user-based tests, such as think aloud 
or field design methods and observation methods (e.g., video observation, screen 
sharing, mouse tracking, eye tracking). Usability evaluation feedback is deployed 
for further development of the system under scrutiny, as they can provide insights 
into where and why usability requirements are not met. 

Think aloud is a method that requires end-users to constantly think aloud as they 
are using a system individually or collaboratively in order to understand how they 
perceive the features of the user interface, identify preferences, and discover any 
potential misconceptions at early design stages (Dumas and Fox 2007). The draw- 
back of this method is that it can be tiring for end-users who have to focus and 
behave in a rather unnatural manner by giving a running commentary on their own 
actions. 

Field methods are a collection of tools and techniques for conducting user 
studies in context. Among others, Contextual Inquiry (Beyer and Holtzblatt 1998) 
is commonly used field method in research as well as in practice. The main 
advantage of such methods is that they provide a development team with data 
about what and how (and why) people carry out their tasks in a given environment, 
thereby enabling the production of useful and usable systems that meet people’s 
needs and goals. The main disadvantage is that they are time-consuming. Nonethe- 
less, such methods can be streamlined with respect to the budget available for 
evaluation in a project (Wixon et al. 2002). 

Furthermore, while the importance of automated monitoring techniques was 
already highlighted above, methods such as CAM and Google Analytics may not 
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provide sufficient granularity of data to determine the usability of the PLE software. 
The ability of CAM to provide granular and contextual data may be useful, but its 
appropriateness may not be established unless or until a sufficient amount of data 
has been collected. Apart from traditional methods mentioned above, there are two 
additional methods that can be useful for small-scale (eye tracking) and large-scale 
(mouse tracking) usability evaluations: 


— Eye tracking measures visual attention as people navigate through websites. It is 
useful in quantifying which sections of an interface are read, glanced at, or 
skipped/ignored. Eye tracking is generally carried out in laboratories and at a 
small scale. It can provide useful information for evaluating the effectiveness of 
the learning design (Schwonke et al. 2009; van Gog and Scheiter 2010) and it 
can be used to gather data after every redesign phase before large-scale rollout. 

— Mouse tracking is a technique for monitoring and visualizing mouse movements 
on any web interface. Mouse movements provide key data about usability issues 
on a large scale, as users can be observed in their natural habitat in an unobtru- 
sive and continuous manner. In most cases, a JavaScript code snippet is inserted 
to track mouse movements. Privacy issues must be considered while adopting 
this method. Tools like Crazyegg,* Userfly,” and Simple Mouse Tracking® can 
be used for this purpose. It should be mentioned that even more so than eye 
tracking, data captured with this method represent only part of the story and, 
hence, must be triangulated with other qualitative data to ensure completeness 
and correct interpretation. 


For summative usability evaluation, user surveys are deployed. They are nor- 
mally administered in the final phase of a project after end-users interact with an 
executable prototype. Among others, the System Usability Scale (SUS) is widely 
used in research and practice, as it is simple with only ten items and standardized 
with psychometric properties (Brooke 1996). 

To study the usage of PLEs, it is crucial to evaluate whether the associated 
services and features can help achieve learning objectives. This can be derived from 
evaluation metadata such as ratings, bookmarks, tags, and comments provided by 
users (Vuorikari and Berendt 2009): One important aspect here is to investigate 
how the PLE usage facilitates social interactions, triggers discussions, and 
improves the understanding of learning content (Mason and Rennie 2007; Farrell 
et al. 2007; Rollett et al. 2007). Moreover, when it comes to learning material 
recommended by the system, ratings and like/dislike evaluation metadata can help 
assess unobtrusively to what extent learners deem them useful. 


4 http://www.crazyegg.com/ 
5 http://userfly.com/ 


6 http://smt.speedzinemedia.com/smt/ 
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User Experience 


The literature on UX published since the turn of the millennium indicates that there 
are two disparate stances on how UX should be studied (i.e., qualitative versus 
quantitative) and that they are not necessarily compatible or can even be antago- 
nistic. A major argument between the two positions is the legitimacy of breaking 
down experiential qualities into components, rendering them to be measurable. A 
rather comprehensive review on the recent UX publications (Bargas-Avila and 
Hornbæk 2011) identifies the following observations: UX research studies have 
hitherto relied primarily on qualitative methods; among others, emotions, enjoy- 
ment, and aesthetics are the most frequently measured dimensions; the products and 
use contexts studied are shifted from work to leisure and from controlled tasks to 
consumer products and art; the progress on UX measures has thus been slow. 

Given that UX has at least to some extent developed from usability, it is not 
surprising that UX methods and measures are largely drawn from usability (Tullis 
and Albert 2008). However, the notion of UX is much more complex, given a mesh 
of psychological, social, and physiological concepts it can be associated with. 
Among others, a major concept is emotion or felt experience (McCarthy and Wright 
2004). As emotion arises from our conscious cognitive interpretations of 
perceptual-sensory responses, UX can thus be seen as a cognitive process that can 
be modeled and measured (Hartmann et al. 2008). 

Larsen and Fredrickson (1999) discussed measurement issues in emotion 
research with reference to the influential work of Ekman, Russell, Scherer, and 
other scholars in this area. More recent work along this direction has been 
conducted (cited in Bargas-Avila et al. 2011). These publications point to a 
common observation that measuring emotion is plausible, useful, and necessary. 
However, like most, if not all, psychological measurements, they are only approx- 
imations (Hand 2004) and should be considered critically. Employing quantitative 
measures to the exclusion of qualitative accounts of user experiences, or vice versa, 
is too restrictive and may even lead to wrong implications (Law et al. 2014). 

There exist a range of UX evaluation methods (e.g., Vermeeren et al. 2010). For 
qualitative data, narrative or storytelling methods (e.g., Riessman 2008) are com- 
monly employed. For instance, users’ short descriptions about their positive and 
negative interaction experiences can be analyzed with the use of machine learning 
as well as manual coding approach (e.g., Tuch et al. 2013). For quantitative data, 
validated scales with good psychometric properties such as AttrakDiff2 
(Hassenzahl and Monk 2010) and PANAS (Positive Affect and Negative Affect 
Scale; Watson et al. 1988) are increasingly used. 

Especially challenging is to operationalize a diversity of emotions, be they 
positive and negative, because teasing out their nuances proves difficult. Common 
methods here are self-assessment manikins and Emocards (for a summary, see 
Stickel et al. 2011). It is even more demanding to measure the social aspect of 
UX, which has hitherto been defined as highly individual and contextualized (Law 
et al. 2009). 
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Organizational Aspect 


With their capability for personalization and plasticity, PLEs help create a rich and 
diverse learning technology ecosystem promising perpetual change and innovation. 
The uptake and effects of PLEs at an organizational level can be understood in the 
light of theory of Diffusion of Innovation, which is advanced by Rogers (1995): 
“An innovation is an idea, practice, or object that is perceived as new by an 
individual or other unit of adoption” (p.11). 

Furthermore, Rogers (1995) states that the “innovation diffusion process” pro- 
gresses over time through five stages: knowledge (when adopters learn about the 
innovation), persuasion (when they are persuaded of the value of the innovation), 
decision (when they decide to adopt it), implementation (when the innovation is put 
into operation), and confirmation (when the decision is reaffirmed or rejected). 

The ROLE project conducted a study to identify factors that can have an effect 
on the adoption and diffusion of PLE-related technologies in organizations (Chat- 
terjee et al. 2013). Table 2 presents an overview of the factors identified. 

Among the main organizational factors, the outlook of the top management on 
introducing technological change matters, as this particularly influences persuasion 
strategies for facilitating positive decision-making in terms of PLE adoption. It is 
equally important to look at how coherent or unified the views on PLEs of the key 
stakeholders within the organization are. With the increasing popularity of social 
media within commercial organizations, extensive use of such platforms can have 
positive impacts on informing the stakeholders about key concepts and issues 
around PLEs. 

The top management, as per the findings of the study, is particularly interested in 
the cost-effectiveness PLEs offer as compared to existing solutions in place—the 
perceived cost-effectiveness thus plays a key role here for evaluation. Compatibil- 
ity with the existing technical infrastructure and high learnability are other key 
success factors of introducing innovation. These persuasive factors tend to act in a 
push-pull mechanism (Shih 2006) before embarking on the decision-making stage. 


Table 2 Potential factors influencing organizational uptake 


Categories Factors 


Organizational Leaderships attitude towards change 


Strategic alignment 


Learning culture 


IT support 


Innovation (PLE) Perceived cost-effectiveness 
Compatibility with existing system 


Perceived effort expectancy 


External factors Perceived factuality 


Communication channels and influence Line manager 


Social networks 
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Once the key stakeholders within an organization are informed and persuaded about 
the usefulness and utility of PLEs within their organization, the top management 
may then take the two key factors into account when deciding upon the adoption of 
the new learning technologies. 

PLEs enable the learners to take control of their own learning depending on their 
contextual needs and goals. It is therefore crucial to check whether a framework 
exists that allows relating personal goals directly to organizational goals. Similarly, 
the learning culture should not be dominated by didactic and trainer-facilitated 
approaches, as a healthy sign of PLE adoption is that learners take control of their 
own learning and managing the related technologies. It is necessary to look at the 
provision of IT support (particularly in the introduction phase), when stakeholders 
start using PLEs within their day-to-day activities. Another important factor that 
determines the PLE adoption is its use by line managers. If line managers and senior 
team do not lead by example, then the likelihood of PLE adoption can be adversely 
affected. 


Psycho-pedagogical Aspect 


From the psychological and pedagogical perspective, the key aspects to look at are 
the ability to foster self-regulated learning, the guidance and recommendation 
strategy, and the facilities for reflection and monitoring. Moreover, the availability 
and documentation of an activity and skill model play an important role—and how 
far this is put into practice. 


Self-regulated Learning 


From the psycho-pedagogical perspective, effective exploitation of PLEs, which 
support lifelong learning, hinges crucially upon the learner’s self-regulated learning 
competence. The quality of learning outcomes varies with the extent to which 
learners are capable of regulating their own learning (Steffens 2006). Self-regulated 
learning approaches have been evolving since the 1970s in educational research and 
practice (Efklides 2009). 

Successful deployment of PLEs relies on a self-regulated learning process model 
such as the following one (derived from Zimmerman 2002), where it is seen as a 
learner-centric cyclic model consisting of four recurring learning phases: learner 
profile information is defined or revised; learner finds and selects learning 
resources; learner works on selected learning resources; and learner reflects and 
reacts on strategies, achievements, and usefulness. 

Note that while cognitive learning activities are rather related to actual learning 
(i.e., information receiving, debating, and experimenting), meta-cognitive learning 
activities are related to controlling and reflecting on one’s own learning. 
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With respect to the evaluation of the success and extent of self-regulated 
learning, gathering data about the accuracy and usefulness of the learning process 
model is crucial. It is particularly relevant to find out, whether learners can actually 
follow the process model and whether they comprehend it and its implications. 
Another key question is, whether the process model supports the development of 
self-regulatory skills. 

It should be taken into account that the process model can be applied in different 
contexts and situations. For example, learners might be in a collaborative learning 
situation, where they may learn together with peers. Or they may learn on their own. 
In addition, the actual learning technology mix may make a difference, since 
learners might use tools and widgets explicitly built to support self-regulated 
learning, whereas in other cases, performance of meta-cognitive learning activities 
may happen just in an implicit way (i.e., being aware of them). 

One particularly useful instrument to help in the evaluation of self-directed 
learning is the questionnaire. While it certainly is supportive of all other aspects 
mentioned above and following below, this widely used instrument can help here in 
providing structured, often numerical data. Questionnaires can be administered 
without the presence of the researcher, and are often comparatively straightforward 
to analyze (Wilson and McLean 1994). According to Cohen et al (2000), “Though 
there is a large range of questionnaires that one can use, but there is a simple rule of 
thumb to follow: the larger the size of the sample, the more structured, closed and 
numerical the questionnaire may have to be, and the smaller the size of the sample, 
the less structured, more open and word based the questionnaire may be” (p. 247). 
Questionnaires are particularly useful when comparison across groups is required 
(Oppenheim 1992). 


Guidance and Recommendation Strategies 


Guidance for learning in the context of PLEs depends on the situation and on who is 
providing the guidance. Learners can learn in a blended learning situation with 
teachers structuring the learning process. Peers can be involved in the learning 
process, if learners collaborate in some way. Learners can also learn on their own 
without human interaction. In the first case, teachers can provide guidance. In the 
second case, peers can provide guidance either directly or indirectly (e.g., with 
peers attempting to master a problem together). In all cases, guidance can also be 
provided by the system through personalized recommendations. 

Moreover, the scope of guidance can focus on a variety of things, including the 
search for learning resources (e.g., widgets, content, or peers), the composition of a 
PLE, the control over the learning process, and the improvement of self-regulation 
ability. Evaluating the effectiveness and appropriateness of such guidance strate- 
gies requires looking into its preconditions: the given abilities of learners are 
relevant, since it depends largely on concrete skills of learners, what they can do 
on their own and where they need help. 
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Furthermore, goals and preferences need to be investigated because the scope of 
guidance depends on these factors. It should be noted that it depends on who is 
delivering guidance, whether certain preconditions can be taken into account, and 
to which extent. If the system provides guidance, then this is done usually in terms 
of recommendations. Personalized recommendations are based on a learner model 
(e.g., goals, skills, learning history, learning progress, background of a learner, and 
the learner’s preferred instructional technique), which models the preconditions for 
guidance. 

The scope of recommendations can include concrete widgets, content resources, 
peers, learning activities, and complete learning environments (i.e., sets of learning 
resources). By recommending certain meta-cognitive learning activities, guidance 
for self-regulated learning can be provided. In case of teacher guidance, learning 
environments can be pre-configured. Especially in a blended learning situation, 
teachers can support the use of the learning environment and help improve self- 
regulated learning, providing further scaffolds to system guidance. 

Regarding evaluation, it is important to assess the appropriateness and quality of 
guidance strategies. This includes evaluating, whether the respective guidance 
strategy helps learning effectively and whether the guidance provided helps over- 
come difficulties. Different guidance strategies have different purposes: it requires 
an evaluation of whether all purposes are actually achieved. 

While of course the questionnaire (see above) can be utilized to evaluate the 
success of particular guidance and recommendation facilities in their context, other 
qualitative methods are suitable as well—such as focus groups, the nominal group 
technique, and a Delphi study. Quasi-experiments using test collections and statis- 
tical measurements are the dominant quantitative methods. 

A focus group is a small group of people who get together to discuss a certain 
issue given to them normally by a researcher. It usually consists of 6-10 members 
and meets regularly during the lifetime of a project or in an ad hoc manner when a 
need arises (Vaughan et al 1996). The technique relies on interactions among group 
members. Focus groups are used to capture qualitative feedback to triangulate 
findings from some other data sources. 

Two other techniques, namely Nominal Group technique and Delphi technique 
may be used to collect group opinion. The Nominal Group Technique was devel- 
oped by Delbecq and Van de Van (1971, 1975) in the 1970s. It has been found to be 
useful in improving educational programs (Jones and Hunter 1995). There is further 
evidence in the literature that it was successfully used for evaluation purposes in 
higher education (Nisbet and Watt 1984). Grant et al. (2003) used the technique to 
determine the impact of student journals in postgraduate education. 

The Delphi technique (Turoff 1970) is, like the Nominal Group technique, a 
structured process, but it does not require physical proximity among participants. 
The participants may be geographically dispersed and are not required to meet face 
to face. Either technique may be instantiated after validation trials to gather group 
data, augmenting and triangulating the monitoring or survey data. 

Following the tradition of search engine evaluation, the relevance of recommen- 
dations can be evaluated in the so-called quasi-experiment with the help of a 
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specially prepared test collection. In such a case, the learning resources (e.g., 
content, peers) are evaluated by experts or representative users; this allows com- 
paring how well the recommender system performs in bringing up the most relevant 
and most complete recommendation. Evaluation measures depend on the guidance 
strategy: for example, recommendations fostering serendipity have much more 
relaxed requirements on accuracy as compared to identifying potential peers who 
are currently in a similar learning situation. An overview on possible evaluation 
measures (and their application contexts) can be found in Herlocker et al. (2004). 


Reflection and Monitoring 


Learner information is important for guidance strategies; this can be the assessment 
of a teacher, peers, or the learner herself. A teacher and peers might form an opinion 
by observing, the learner can do this by self-monitoring or self-reflection, and the 
system can do that by tracking the learner’s behavior and building a learner profile 
(or recommending profile information). Most importantly, a mixed procedure can 
be used if profile information is proposed by the system and the learner has to 
modify and update it. In this case the learner is made aware of certain assessment 
outcomes, which also stimulates self-reflection. As already mentioned above, 
learner profile may contain information about goals, skills, learning progress, etc. 
Evaluation should focus on the accuracy of this information. 

While an interview can be used for the evaluation of many of the other aspects 
listed above and below, it is particularly useful for the evaluation of reflection and 
monitoring. An interview is a purposeful discussion between two or more people 
(Kahn and Cannel 1957). One of the most distinct advantages of interview over, for 
instance, questionnaires is that the researcher has personal contact with the respon- 
dent and hence more control over the questions and its context. The researcher is 
available to clarify confusing questions (Cohen et al 2000), which is difficult to do 
with questionnaires. This same advantage, however, can also turn into a disadvan- 
tage, when the researcher knowingly or unknowingly diverts the discussion and 
when allowing personal bias to directly impact on outcomes. Interviews consist of a 
more direct method that helps easily spot user preferences, satisfaction, and 
encountered problems. 

Apart from qualitative approaches, quantitative evaluation techniques utilizing 
content analysis over learners’ writings are emerging, some of which using auto- 
mation techniques from text mining and statistical processing. Ullmann 
et al. (2013) provide an overview and a framework for the study of reflection by 
hand and with the help of automation techniques; from natural language processing 
as well as using crowd-sourcing of human coding on platforms such as 
CrowdFlower or Amazon’s Mechanical Turk. 


A Multidimensional Evaluation Framework for Personal Learning Environments 65 


Activity and Skill Model 


For successful deployment of PLEs, the underlying skill model is typically com- 
plex, since in addition to the developed domain knowledge, self-regulated learning 
and the handling of PLE services and tools have to be considered. Any PLE skill 
model encompasses at least these three different kinds of skills: domain, tool, and 
self-regulation skills: 


— Domain skills are skills that a learner possesses, if he or she has a certain level of 
expertise in a knowledge domain. For instance, the learner can explain what 
percentages she estimates to have attained and, if she prefers, justifies with some 
qualitative comments. 

— Tool skills are defined as skills which a learner possesses, if she is able to 
perform a learning activity with a learning tool in a domain context: for example, 
the learner can use a tool for setting goals or can use a tool in order to retrieve 
domain knowledge in a certain topic. Different learning activities with the same 
tool can require different skills. 

— Self-regulated learning skills imply the ability of a learner to regulate her 
learning activities by herself: the learner can realistically set own goals, monitor 
own progress, apply effective time management, and self-evaluate. Self-regu- 
lated learning skills are skills on a meta-level and domain independent. 


For the evaluation, focus should be set on documenting and subsequently 
assessing accuracy and usefulness of these skill models. Methods for the assessment 
of accuracy and usefulness are essentially the same as those valid for evaluating the 
utility of PLE utility (particularly automated monitoring and CAM). 


Social Aspect 


A Community of Practice approach is an effective way of sharing knowledge. They 
are usually characterized by anonymity and an addictive, but voluntary behavior, 
with a strong sense of belonging (Hampton and Wellman 2001). Trust, loyalty, and 
social usefulness are pertinent motivational features identified in the virtual com- 
munity context. 

Over the last century, a number of motivational theories were proposed (e.g., 
Maslow 1954; Herzberg 1987; Vroom 1964). At the foundation of these theories, it 
is claimed, lies the suggestion that each school of thought focuses on certain factors 
to the exclusion of all others—for example, reward, social needs, or psychological 
growth. 

A few key inferences in the context of PLEs from the motivational models are 
mentioned below: 


— Recognition of a range of individual needs: Learners have varying levels of 
motivation depending on their needs. 
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— Goal alignment in the provision of materials: If a given task does not align with 
the learner’s goal, then the motivation to complete the task will obviously 
decrease. 

— Varying incentives: Incentives can help instill a sense of achievement and 
motivation to keep going. Learners will require varying levels of incentives of 
different natures to keep themselves motivated (grades, peer recognition, altru- 
ism, to mention just a few). 

— Connectedness to community performance: Link of these incentives to perfor- 
mance at an organizational or community level. 


To assess the social aspect of PLEs, Kim’s (2000) application of Maslow’s 
Hierarchy of Needs to online communities can be further adapted: Table 3 


Table 3 Community building and motivation (extended from Kim 2000) 


Need 
Physiological 


Offline Online communities (Kim Personal Learning 

(Maslow) 2000) Environment 

Food System access Access to PLE 
technology, widget store, 
user profile 

Clothing Ability to maintain own identity | Use of templates for 

Shelter while participating in the assembly of environment 

health community 


Security and 
safety 


Protection from 
crimes and war 


Protection from hacking and 
personal attacks 


Data security (automated 
monitoring data) and 
encryption 


Sense of living 
in a fair society 


level playing field 


maintain varying level of 
privacy 


multi-level privacy 
framework 


Social 


Self-esteem 


Give and 
receive love 


Feeling of 
belongingness 


Self-respect 


Ability to earn 
respect from 
others 


Belonging to the community as 
whole and within subgroups 


Contribute to community and 
get recognized 


Share and consume tools, 
content, and resources 


Belongingness 


Ability to collaborate 
across several social 
networks 

Sharing modified PLE 
templates 


Altruism 


Mentoring 


Giving and receiving 
feedback 


Rating and ranking 


Self- 
actualization 


Develop skills 
and fulfill one’s 
potential 


Take on community role that 
develops new skills and opens 
new opportunities 


Acquiring expert status 
within the community 


Assembly and regulation 
of own learning 
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illustrates which constructs are relevant to the PLE evaluation from a motivational 
perspective. 

Clustering techniques and social network analysis (SNA) can be used to trace 
whether the infrastructure supports the emergence and evolution of self-directed 
communities of interest and practice (Wenger 1998). Both rely on either implicit 
factors (looking at interaction and usage patterns) or explicit ones (utilizing eval- 
uation metadata). 

SNA originates from sociology and network analysis that is widely applied in 
physics, electrical science, civil engineering, and others. In SNA, entities and 
relations among them are mathematically modeled as graphs, (i.e., sets of nodes 
and edges connecting them). Nodes and edges can have different semantics: for 
instance, nodes can be people and edges between nodes can be based on commu- 
nication between people, for example, through e-mails or chats. Edges can also be 
used to denote citations of resources that peers own or create. For instance, a peer is 
connected with the other one whose work he has cited. According to the Actor 
Network Theory (Latour 1991), we can consider every node as an arbitrary actor, 
which is not necessarily human. In this sense, it is also possible to analyze networks 
consisting of users and tools, both modeled as nodes. 

SNA is a basis for assessing social learning and the interaction with tools used in 
learning (Klamma 2010). It helps discover information about social relationships. 
Based on this, it allows inspecting social presence of learners within their commu- 
nities: for example, it helps in evaluating which roles learners adopt or how their 
positions evolve over time, positively as well as negatively. 

Since 1967 with the discovery of the small world network phenomenon (Milgram 
1967), the heterogeneity of networks has been examined intensively. Newman (2003) 
showed that in scale-free networks, connections between nodes are distributed 
unequally with a certain probability. While most of the nodes have few connections, 
there exist a few nodes exhibiting a large number of connections. The connectivity of a 
graph representing a network informs about robustness and cohesiveness of the 
network (Brandes and Erlebach 2005). Freeman (1979) also pays attention to centrality 
measures that help us to reveal special roles of network nodes. Moreover, brokerage 
phenomena can hardly be defined without the application of SNA (Barabasi 2007). 
Considering the irregularity of peer connections of networks, Newman and Girvan 
(2004) developed one of the clustering algorithms, which find groups of network nodes 
that are densely connected to each other but sparsely connected with other nodes. 

Networks typically consist of several groups of learners communicating with 
each other and with other groups. SNA techniques and clustering allow unveiling 
the structure underlying such a network. For example, networks can include groups 
of learners that have connections only to leaders of groups, but don’t have com- 
munications with other groups. 

SNA techniques allow following behaviors of learners within a time frame by 
examining network centrality measures, which reveal expertise or presence of a 
learner within a network. This method of evaluation may show us how learners 
evolve in their communities over time: do they become experts or brokers of 
information from one to the other community or do they lose their position and 
lock themselves in a community closed from communication? 
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In practice, SNA requires the availability of data containing information on the 
nodes, i.e., people, groups of people or even tools, and on the edges, i.e., relations 
between nodes. One possible source of input for SNA can be the raw monitoring 
data. Here, different kinds of interaction between users are captured. 


The Unified PLE Evaluation Framework 


Based on the TOPS model and the background literature reviewed above, we 
propose an integrated evaluation framework for PLEs. Specifically, the framework 
incorporates major dimensions with a gradual progression from the individual to 
community focus. Figure 2 lists the key dimensions (and its aspects) of this 
evaluation framework and shows how they relate to each other: the framework is 
organized in three circles from the inner Technological one, which lays the cornerstone 
of PLEs, through the middle Psycho-pedagogical circle, which addresses individual 
user’s needs and goals, to the outer Organizational and Social circle, which brings in 
the social and organizational factors relevant to the exploitation of PLEs. 

The constructs highlighted within the three circles are high-level concepts, 
which should be translated into low-level variables, selected from the review 
brought forward in the previous sections. Operationalizing and estimating such 
variables with particular techniques and tools leads to results, which can somehow 
and somewhat account for the extent to which PLEs successfully enable users to 
attain their learning goals. For instance, the construct usability is translated into two 
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metrics—effectiveness and efficiency— which can be measured in terms of number 
and type of errors and in the time to complete a specific task with a PLE. 

Nonetheless, not every construct can be operationalized in a straightforward 
manner. Indeed, it is a challenging task to develop structural and measurement 
models, where factors and measures are orthogonal in the ideal case, but at least 
exhibit a lowest degree of collinearity. Statistical analysis techniques such as 
correlation, regression, and factor analysis deem useful to sample, validate, and 
tune the underlying model in early evaluation runs in order to maximize validity 
throughout the overall process. 

Table 4 relates these three sets of dimensions (with their main criteria) to the 
methods reviewed in the previous sections. Each of the dimensions (technological, 
psycho-pedagogical, and organizational/social) can be broken down into its main 
groups of constructs, as listed in the first column. The second column provides the 
selection of methods that have been used in the past and that we deem most 
appropriate for their study. 


Table 4 Evaluation dimensions and recommended methods 


Group of 
Dimension constructs Key methods 
Technological | Openness Questionnaires 
Responsiveness Interviews (incl. storytelling) 
Security Desk research (documentation) 
Scalability Nominal group, Delphi 
Documentation Inspection methods 
Interoperability User tests 
Accessibility Monitoring data (incl. web analytics, CAM) 
Availability, Observation 
reliability 
Quality (content Unit tests 
and UI) 
Effectiveness user-based evaluation: behavioral measures, observa- 
Efficiency, tions, and questionnaires 
satisfaction 
Enjoyment 
Organizational/ | Trust Questionnaires 
Social Social usefulness | Interviews 


New connections Desk research 


Sharing Monitoring data 
Privacy Social network analysis 
Clustering 
Psycho- Meta-cognitive Questionnaires 
pedagogical Motivation Interviews 
Behavioral Nominal group, Delphi, focus group 


Recommendations | Monitoring data 


Observation 


Quasi-experiments (relevance accuracy) 
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The PLE evaluation is ideally conducted in cycles of planning, actual evaluation, 
and reflection on results. A useful vehicle for this can be found in form of case 
studies and—concluding the final cycle—a cross-case analysis. Case study is a 
generic term for the investigation of an individual group or a phenomenon (Bogdan 
and Biklen 2006). Case studies are often used for exploratory research, but the 
technique can be varied and adapted to include the multi-method mix proposed 
above for the unified PLE evaluation framework. 

While the techniques used may vary, the distinguishing feature of case study is 
the assumption that human systems develop a characteristic wholeness or integrity 
and are not simply a loose collection of traits. This approach enables researchers to 
investigate a given phenomenon to a much greater depth, bringing out the interde- 
pendencies of parts and emerging patterns. Besides, case study has the potential to 
accommodate the value context of the enquiry, is flexible to accommodate unan- 
ticipated events, does not attempt to generalize, and admit the problems of 
researcher bias in various ways (Nisbet and Watt 1984). Nonetheless, the inability 
to accommodate re-observation is a major cause of concern. 

The final cycle of the cyclic evaluation process depicted above in Fig. 3 can then 
be concluded with the cross-case analysis. A cross-case analysis is “a qualitative, 
inductive, multi-case study that seeks to build abstractions across cases” (Merriam 
1998, p.195). It is used to identify and compare patterns of similarities and 
differences across individual cases resulting in meaningful connections. Most 
importantly it empowers all stakeholders to access new knowledge from a rich 
holistic point of view (Khan and van Wynsberghe 2008). 

There are two well-known techniques to carry out cross-case analysis, namely, 
variable- and case-oriented approaches (Ragin 2004). There are other techniques as 
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Analysis 


Reflect 


Evaluate (Re-)Plan 


Fig. 3 Evaluation cycle for PLEs 
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well but are generally derived from the aforementioned ones. The variable-oriented 
technique focuses on comparison of identified variables across cases in order to 
delineate causal relationships. The case-oriented approach enables researchers to 
make sense of causal similarities between different cases by comparing them using 
visualization techniques such as stacking cases (Miles and Huberman 1994), 
thereby enabling the identification of new social phenomenon. 

There are a number of ways in which case-oriented cross-case analysis could be 
carried out, namely, most different design (Przeworski and Teune 1982), typolo- 
gies, multi-case methods (Smith 2004), and process tracing (George and Bennett 
2005). The first two are of particular interest for PLE. The aim for adopting cross- 
case analysis for studying the implementation of PLEs across settings is to identify 
similarities in a diverse set of cases, which is what most different design offers. 
Additionally clustering of cases might also be relevant to identify and compare 
patterns and process pathways to seek typological regularity. We recommend the 
adoption of an iterative case study design with multi-method data collection to 
triangulate empirical findings. Cross-case analysis should be performed towards the 
end of a series of evaluations to obtain a holistic view on the outcomes of deploying 
PLEs (cf. Fig. 3). 


General Discussion: Qualitative Versus Quantitative 


In the foregoing sections we present an array of quantitative and qualitative 
methods for data collection and analysis. The selection of a particular type of 
method depends on individual researchers’ assumptions, values, and expertise. 

Some researchers defy the value of quantitative data with the argument that 
numbers cannot tell us anything, insisting on capturing solely qualitative data. Any 
method fundamentalism is wrong, not least in the light of a postulate for a wide 
repertoire of research skills among researchers. Still such standpoint is often found 
in practice, particularly by those critics instigating methodological discussions with 
the aim to dismantle or even discredit a particular piece of quantitative work they do 
not agree with. 

It is in our opinion, however, not that simple: Methods cannot be differentiated 
into good and bad, and if a particular method fails to provide results (or even more 
often: results beyond tautologies), then this probably says more about their com- 
petent handling, rather than their validity or reliability. Exceptions prove the rule, of 
course. 

In our view, there are two aspects to consider that influence methodological 
choices. First, it all depends on why the evaluation is needed, what the goal of the 
evaluation is, and who the recipient of the evaluation data is. For example, if the 
target is to feed back into psycho-pedagogical or technological development, 
qualitative means can provide deeper insights on what has gone wrong, what 
works, and what leaves room for improvement. Moreover, qualitative methods 
bear the potential to discover, why this is the case. 
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Furthermore, which approach to adopt depends on the phase of a research study. 
Qualitative approaches are particularly useful for exploring a topic and its phenom- 
ena in their context. They help in forming hypotheses and build understanding. 
Once such understanding is reached, however, more targeted questions can be 
posed. Also, if a phenomenon or an application is potentially relevant to a larger 
number of people, then it is well justified to conduct a quantitative follow-up to see 
if the qualitative findings, suspected dependencies, effects, and other observations 
hold when scaling out. Qualitative methods do not scale very well, which can pose a 
problem when the target is to, for instance, to assess the effects of an intervention on 
a full university, an entire company, or the general population. 

This chapter aims to support researchers in determining which method they 
need, depending on purpose (“TOPS”) and phase (from case-to-case to cross-case). 
It provides a rich repertoire of different methods for the multi-method, multi- 
perspective mix, and it helps in combining the strength of different approaches 
into a unified evaluation. 

As can be seen from the review of the methodological state of the art, the 
frontiers in technology-enhanced learning are much more complex than the mere 
differentiation of quantitative and qualitative suggests: “mediated” observation 
using monitoring data, pictogram-based methods for affect measurement, quasi- 
experiments for relevance evaluation, and the like start blurring these boundaries 
and start claiming their own place in the standard canon of methods. 

It is worth mentioning one class of methods listed in the chapter in particular, as 
it stands out through the paucity of research in the area of PLEs: While emotions 
and affects can play a critical role in influencing a learner’s motivation to engage in 
technology-enhanced learning activities, this experiential aspect tends to be not 
only overlooked, but also under-researched. 

At the turn of millennium, the psychological research on emotions has been 
rekindled, thanks to the work of psychologists such as Klaus Scherer (2005; 
“emotion wheel”) and James A. Russell (2003“core affect”). Coincidentally, this 
resurgence of interest in emotions and affects has resonated with the shift of 
emphasis in HCI around the same time, moving from cognitivist-behavioral per- 
formance-based usability to phenomenological-reflective experience-oriented user 
experience (UX) (Law et al. 2009) . 

Alongside with this change of emphasis is the revived tension about the relative 
importance of qualitative and quantitative methods. This issue is actually an 
age-old debate in the realm of measurement theory. In brevity, some UX 
researchers argue that experience is holistic and cannot be reduced into components 
to be measured; any attempt to put down a number to infer the type or intensity of an 
emotion is methodologically flawed and inherently meaningless. In contrast, some 
other UX researchers believe that the process of experiencing/experienced emo- 
tions can be modeled like cognitive processes and thus they are measurable. These 
arguments have significant implications to the selection of evaluation methods for 
assessing the impact of interacting with technologies (Law et al. 2014). 

Above all, putting aside the issue about the quantifiability of user experience, the 
main point we want to stress is the high relevance of emotions and affects to the 
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design and evaluation of learning environments. Both positive (e.g., fun, pleasure, 
engaged, liberating) and negative (e.g., anxious, defeated, frustrated, fear) emotions 
can substantially shape the effectiveness of any type of learning situations, includ- 
ing PLEs. Consequently, due attention should be heeded to this overlooked expe- 
riential aspect. 


Conclusion and Future Work 


Developing an evaluation framework for PLEs is challenging, since technological, 
organizational, psycho-pedagogical and social aspects need to be considered in an 
integrated manner and with a diverse set of stakeholder perspectives being taken 
into account. 

Our attempt was to propose a unified framework encompassing the main valid 
constructs (derived from relevant theoretical models), yet at the same time provid- 
ing a flexible and adaptive methodology that is capable of accommodating the 
changes that are inevitable in an emerging field. 

In order to achieve this, we have elaborated an integrated framework that is by 
nature case study based and follows a multi-method approach. Furthermore, we 
recommended concluding the cyclic evaluation with a cross-case analysis in order 
to consolidate data from different contexts so as to establish a holistic view. 

A number of metrics and possible methods have been identified and located in 
the proposed unified framework. The metrics, criteria, methods, techniques, and 
tools proposed are subjected to further refinement and improvement. A process 
model ensures the possibility to do so in a well-defined manner. 

Obviously, more research efforts are called for to investigate the complex 
phenomenon of PLE—and this contribution provides the methodological basis on 
which such future endeavors can be built. 
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Abstract Formal learning in higher education creates its own challenges for 
didactics, teaching, technology, and organization. The growing need for well- 
educated employees requires new ideas and tools in education. Within the ROLE 
project, three personal learning environments based on ROLE technology were 
used to accompany “traditional” teaching and learning activities at universities. The 
test beds at the RWTH Aachen University in Germany, the School of Continuing 
Education of Shanghai Jiao Tong University in China, and the Uppsala University 
in Sweden differ in learning culture, the number of students and their individual 
background, synchronous versus distant learning, etc. The big range of test beds 
underlines the flexibility of ROLE technology. For each test bed, the learning 
scenario is presented and analyzed as well as the particular ROLE learning envi- 
ronment. The evaluation methods are described and the research results discussed 
in detail. The learned lessons provide an easy way to benefit from the ROLE 
research work which demonstrates the potential for new ideas based on flexible 
e-learning concepts and tools in “traditional” education. 
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Introduction 


This chapter focuses on using Personal Learning Environments (PLEs) in formal 
learning in higher education (HE). Here, formal learning means that the PLE and its 
widget bundles support the established, “traditional” way of teaching in a lecture. 
The teaching and learning activities are not newly created and centered on the PLE, 
but the PLE extends the existing teaching context and provides additional activities 
within. Therefore, the primary audience of PLEs consists of teachers instead of 
students. While this might sound as a contradiction to the paradigm of personal 
learning, we argue that a valuable goal of ROLE technology is to increase the range 
of interactive and social learning opportunities. 
In the ROLE project, three test beds served to explore such a setting: 


e RWTH Aachen University, whose department of mechanical engineering is 
ranked at 17th in the world (best in Germany) by the QS University Subject 
Ranking in 2012. 

¢ The School of Continuing Education (SOCE) of Shanghai Jiao Tong University 
(SJTU), a blended learning institution whose students are young, working adults 
who study part-time. 

¢ Uppsala University, Sweden’s oldest university founded in 1477, which has got 
a long tradition of distance education. 


Albeit all three test beds are placed within higher education, they cover quite 
different contexts. For instance, SOCE is located in China. With its approach to 
teaching and learning based on the Confucian tradition (Zhang 2007), it is quite 
different to RWTH as an example of a Western university. Also, SOCE students 
study part-time, most of them have a job and family, while RWTH students are 
younger full-time students. Furthermore, SOCE is a blended institution, where a 
significant part of teaching and learning takes place online, while RWTH is a 
traditional on-campus university. In contrast, while Uppsala University is also for 
the most part an on-campus university, it offers a wide selection of distance courses 
with very limited or no on-campus participation required. It proved particularly 
interesting to investigate what different forms ROLE technology could take in these 
settings, and it speaks of its flexibility that this was possible at all. Last but not least, 
the number of participants in the test bed classes differ from 20 (Uppsala Scenario) 
and 250 (SOCE Scenario) to 1,600 (RWTH Scenario). Growing numbers of 
students combined with limited resources for teaching are often an important 
motivation to search for a better support by e-learning tools. 

The chapter starts with a review of related work, followed by the description of 
the test beds in separate sections. In each section, we describe the learning context, 
the tools and bundles employed, the most relevant evaluations we performed and 
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the lessons we learned. We end this chapter with a brief conclusion that summarizes 
the main differences and similarities regarding the usage of ROLE technology in 
the three test beds. 


Related Research 


The presented approach addresses various recent research issues such as teaching 
and learning in large classes as well as using cloud services and Web 2.0 applica- 
tions for e-learning support. 

The usage of PLE technology has been investigated by a few studies, albeit in 
small-scale environments. Blees and Rittberger (2009) describe the usage of a 
learning environment assembled from different Web 2.0 services in a course on 
“Social Software.” The 13 participants were familiar with Web 2.0 technology and 
rated the usage of the service relatively high. The 26 case studies reported by 
Minocha in (2009) mainly cover studies where students worked with a single Web 
2.0 service integrated into a PLE. Law and Nguyen-Ngoc (2008) present a social 
network and a content analysis of interactions in a collaborative learning environ- 
ment. Their data show that some students profit from such environments, but not all 
students. The challenge of teaching large classes has been a research issue for many 
years (cf. Leonard et al. 1988; Knight and Wood 2005). The more technical 
background of building e-learning tools from Web 2.0 components is being 
discussed in Palmér et al. (2009). The approach uses six dimensions for the 
mapping of Web 2.0 applications to personalized learning environments. The 
capabilities of ROLE-based cloud learning services are investigated in Rizzardini 
et al. (2012). The evaluation shows that a cloud-based learning support with ROLE 
environments is possible but the learners may need introduction and time to be 
familiar with interactive e-learning tools. The particular aspect of navigation 
guidance for learning questions in Java programming is discussed in Hsiao 
et al. (2010). 

While these studies shed light on specific questions regarding of PLE, no prior 
work has investigated how a single PLE platform can be adapted to suit the needs of 
different formal higher learning scenarios and how it performs in such scenarios 
over long periods of time and with significant number of users. More specifically, 
the case studies described in this chapter add to the mentioned evaluations insofar 
they investigate (1) the use of learning environments that contain components apart 
from Web 2.0 tools (in a narrower sense), (2) with large user groups (3) of both 
teachers and learners (4) from different cultural contexts. The results are ultimately 
relevant for ROLE-based environments but can easily be transferred to other kinds 
of environments and systems. 
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RWTH Aachen University: ROLE for Full-Time Students 
in Large Classes 


Large classes at universities create their own challenges for teaching and learning. 
Audience feedback is lacking. Individual needs of students are often hard to address 
sufficiently. At RWTH Aachen University, a ROLE-based knowledge map learning 
tool was developed and embedded in the context of a large class course for computer 
science in mechanical engineering. The objective of this PLE was to support 
individual learning of students during exam preparation. Theme-based exercises 
have been developed and evaluated. The tool was grounded in the notion of self- 
regulated learning (SRL) with the goal of enabling students to learn independently. ' 


Learning Scenario 


The Institute of Information Management in Mechanical Engineering (IMA) of 
RWTH Aachen University offers a lecture about computer science in mechanical 
engineering, which was attended by 1,600 students in 2012 (see Fig. 1). The lecture 


Fig. 1 Lecture for computer science in mechanical engineering given in the RWTH auditorium 
maximum (made by David Emanuel) 


'Parts of the content and information of this section have already been published by Vieritz 
et.al. (2013). 
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is part of the curriculum for the bachelor degree in mechanical engineering (second 
semester) and business engineering (fourth semester). 

In 2012, the lecture focused on object-oriented software development with Java 
and on software engineering (for details see Ewert et.al. 2011). The lecture is 
accompanied by a programming lab, a group exercise, and exam preparation 
courses. In the lab, the students are taught to program Lego NXT Mindstorms 
robots with Java. They are working in small teams of two students in problem-based 
learning scenarios. They were requested to program a robotic gripper inspired by 
industrial robots. 

The robots simulated pick-and-place machines (P&Ps) as they are used for 
surface-mount devices (SMDs). The resemblance to industrial robots was meant 
to result in a better understanding of mechanical engineering principles by the 
students. To support the Java programming language implementation on the NXT 
controller, LeJOS was used (Solorzano 2012). 

The lab took place together with the lecture during the summer term 2012. The 
lecture period started in April and ended in July. Exam preparation courses were 
provided in September just before the final test. These courses offered the students 
the possibility to train the addressed competences in smaller audiences. 

All parts of the course received good feedback and results from the students 
within the evaluation. Nevertheless, the students were challenged by learning in 
large classes in the programming lab. Individual support was often requested, but 
the number of supporting tutors was limited. 

Therefore, one important objective for the course revision in 2012 was a better 
support for individual learning with e-learning tools. The e-learning system L2P of 
RWTH? is already used as a Learning Management System (LMS) in the lecture, 
the group exercises, and the lab mentioned above. However, additional learning 
support was requested to assist students in and out of class, but particularly when 
learning autonomously. Two major challenges of the described scenario are: 


e A wide range of pre-course programming skills among the students. 
¢ Individual support for learning with limited resources for teaching personal. 


To meet these requirements, a Web-based e-learning test bed was designed and 
implemented which supports different kinds of learning situations like SRL, peer- 
instruction learning, and email support by tutors. The test bed learning content 
ranged from exam preparation exercises for all students to additional background 
information for advanced students. It extends the L2P learning room with interac- 
tive learning capabilities and is described in the next chapter. 


2 http://www?2.elearning.rwth-aachen.de/english 
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The development of the interactive e-learning platform was part of the ROLE 
project. Beginning with the summer semester 2010, a previous version of a Web 
2.0 Knowledge Map (WKM) was enhanced with ROLE technology. In particular, it 
was transferred to a widget-based environment (cf. von der Heiden et al. 2011), that 
is a bundle of widgets interacting via ROLE Inter-widget Communication (IWC) 
(Renzel 2011). 

The WKM is an electronic reference book, which can be regarded as a kind of 
improved Wiki system. It won the second prize in the 2010 International 
E-Learning Association Awards, in the category “Academic Blended Learning.” 
The application supports students in looking up factual knowledge. Students can 
search for articles by entering topic keywords and by navigating from their current 
article to related articles following hyperlinks. It is based on semantic net techno- 
logy, where hyperlinks are not just links, but belong to predefined categories, each 
bearing a meaning, as a named relation. The object-oriented content organization 
knows classes and objects of knowledge. A class is a predefined template for a 
knowledge object such as an “Exercise” and its realization. Similar to a Wiki, the 
WKM supports the creation of new content. A dedicated rights management allows 
the usage of different roles as authors, administrators, and users. Authoring is 
currently restricted to lecturers and tutors. Additionally, the content visualization 
capabilities based on hypermedia support nonlinear learning approaches. 

For the ROLE project, the WKM was redesigned as an interactive learning tool 
and as a test bed for ROLE technology in a higher education scenario. The new 
design was motivated by the following main goals: 


e Guide and support students in a self-regulated and nonlinear learning process. 

e Motivate, introduce, and provide high-quality basic knowledge using multime- 
dia material. 

e Provide an interactive reference book on lecture contents for exam preparation. 

e Support interest-based real-time communication and collaboration in learner 
communities. 


Thus, the former WKM has been extended with a chat to provide theme-based 
learning communication between users. A learning history accomplishes the setup. 
Built up with ROLE technology, the “new” WKM is composed of three inter- 
communicating widgets (see Fig. 2), namely: 


e Web 2.0 knowledge map widget for accessing and reading topic articles as well 
as exam exercises. 

e Chat widget: general or topic-related group chats and presence information for 
individual tutor support or peer-to-peer instruction. 

e History widget: tracks individual learning activities and shows personal history 
of visited topics. 
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Fig. 2 Screenshot of RWTH testbed with Web 2.0 knowledge map, chat and history widget 


The test bed scenario was deployed for the course lab and also for the students’ 
individual exam preparation in August and September. The WKM aimed to provide 
the students with information covered in the lecture and in the lab. It was filled with 
additional SRL-adapted content thus focusing on typical SRL situations such as the 
exam preparation phase. It contained explanations and motivations for notions, 
definitions, or examples, e.g., for basic Java programming constructs. Background 
information was provided as well, e.g., about software installation. Exercises for 
exam preparation were associated with lecture content. The presentation and 
organization of the WKM followed the paradigm of object-oriented analysis and 
design in software development. Relations between objects and classes of objects 
were visualized (see Fig. 3) to underline knowledge associations. Functionalities 
for annotations, remarks, and feedback were provided. 

The second widget, a chat widget, was embedded to offer students the possibility 
to ask and answer topic-related questions. Other students answered the posed 
questions while a tutor moderated the chat. 

Finally, a history widget was embedded into the learning environment. It 
supported the backward navigation within the environment by offering the last 
five activated knowledge objects. The widget uses data from the WKM widget to 
support the learner with his or her own learning history. 

The WKM was maintained by the IMA, the test bed was hosted by the depart- 
ment of information science at RWTH. Access to the WKM was granted via the 
login for the course lab. For the first time in the course’s history, this WKM learning 
environment gave students the opportunity of individual support during their exam 
preparation. 


86 H. Vieritz et al. 


ROLE Knowledge Map 
zur Info1-Projektauigabe 


Programmierung 
7 Object(s) available 


Objektorientierte 


Modellierung 
13 Object(s) available 


leJOS-API 
11 Object(s) available 


Fig. 3 Screenshot of the Web 2.0 knowledge map RWTH (start page) 


Technical realization: The WKM has been bundled with a chat widget and a 
personal history widget. The chat widget allows learners to communicate with 
instant chat messages and to see the online status of other learners. It is integrated 
with the WKM by automatically creating a separate chat room for each topic that is 
currently read by the learner. Learners can see the topics of other learners and can 
quickly join them in the topic-specific chat room to discuss their understanding of 
the topic and how it relates to their current work. The personal history widget 
records the topics visited in the knowledge map and allows quickly navigating back 
to previous topics. The three widgets interoperate based on IWC; the following 
examples of widget communication events demonstrate a selection of implemented 
interactions: 
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Entering a topic-based chat room on topic selection: When a student selects a 
topic from the WKM, a corresponding chat room is entered in the chat widget. 
At the same time, the student’s online status is changed to the new topic in real- 
time and visible to and clickable for other students. 

Following a user’s activity: When a student clicks the online status of another 
student, he navigates to the corresponding topic in the WKM, in turn triggering 
an event to enter the corresponding topic-specific chat room. 

Real-time updates of learning history: When a student selects a topic from the 
WKM, the selected topic appears at the top of his or her personal history. 


Following the overall ROLE approach of open standard compliant widget-based 


learning environments, the WKM test bed was implemented involving the follow- 
ing enabling technologies: 


OpenSocial*: OpenSocial is a set of common application programming inter- 
faces (APIs) for Web-based social network applications developed by Google 
along with MySpace and a number of other social networks. Applications 
implementing the OpenSocial APIs will be interoperable with any social net- 
work system that supports them. The ROLE version of the WKM is deployed in 
Apache Shindig,* the open-source reference implementation of an OpenSocial- 
container. 

Extensible Messaging and Presence Protocol? (XMPP) is an open standard 
technology for real-time communication, which powers a wide range of appli- 
cations including instant messaging, presence, multiuser chat, and collaboration. 
The ROLE version of the WKM offers XMPP-based features such as topic- 
based chat rooms and real-time information on current presence and learning 
activities. 

Inter-widget Communication (IWC) (cf. Renzel 2011; Zuzak et al. 2011): With 
IWC, individual widget functionalities can be combined to realize complete 
application workflows. ROLE leverages various forms of both local and remote 
collaboration and communication among. The ROLE version of the WKM 
demonstrates local IWC using technologies such as PMRPC° and Google Gad- 
get PubSub being part of the OpenSocial specifications. A basic form of remote 
IWC was demonstrated with the new WKM chat functionality. 

Monitoring: All learning activities are tracked by the history widget and 
persisted as Contextualized Attention Metadata (CAM; cf. Schmitz 
et al. 2011). The ROLE version of the WKM was the first test bed producing 
real-life usage data, which, later on, served for producing recommendations and 
as a basis for further development of the WKM and ROLE technologies in 
general. 


7 http://www.opensocial.org/ 


4 http://shindig.apache.org/ 
5 http://xmpp.org/ 
6 http://code.google.com/p/pmrpe/ 
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A detailed description of the ROLE framework technology can be found in 
chapter VIII “Lessons learned from the development of the ROLE framework.” 


Evaluation and Methodology 


Additionally to the tool development, a test bed evaluation was designed to analyze 
how the environment influenced the students’ learning processes. The RWTH 
ROLE test bed work in 2012 was initiated with a Web-based survey that aimed to 
collect details about the students’ experience with e-learning and SRL at the 
beginning of the lab in April 2012. The ROLE widget environment was introduced 
to the students during the second week of their studies. The enriched ROLE-based 
learning environment offered additional support for improvement in SRL opportu- 
nities. It also provided information about programming in general, related tools, 
modeling as well as Java as such. Around 1,600 students participated in the course. 
All students were informed about the ROLE-enhanced learning environment via 
several announcements during lectures and labs as well as via email. During the 
standard midterm teaching evaluation, a short ROLE-related survey was issued. At 
the end of the lecture period, the ROLE test bed was also adapted for individual 
exam preparation during summer time. Finally, after the exam, educational staff 
was interviewed to evaluate the environment and its application within the course. 
The lab sessions took place in the largest computer pool of the RWTH which is 
equipped with approximately 200 workstations. This, however, restricted the max- 
imum number of students that could attend the lab in parallel to 200 students who 
then worked with 100 Mindstorms NXT robots. Since those 100 robots could not be 
dismounted and reassembled in each lesson, the lab was based on a standardized 
and preassembled robot model. 

The ROLE environment was used during the lab time from April to June. Usage 
grew significantly in September when the students started their individual exam 
preparations some weeks before the exam. The access peak was reached in the days 
just before the exam when students switched to “power learning.” This is illustrated 
by Fig. 4 showing the number (by day) of accessed knowledge objects. The number 
of generated views corresponds with the access rate and indicates the intensity of 
usage by the students. Figure 4 underlines the exam-oriented learning during the 
preparation that restricts the leeway in learning and thus the autonomy of the 
learner. This characteristic learning activity trend has been repeated during the 
next exam period in March 2013. 
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Fig. 4 Requested knowledge objects by day in the RWTH testbed 
Results 


In June 2012, before the summer break (i.e., at the end of the lab session but before 
the exam preparation), the students were asked about the usefulness of the 
e-learning environment and rated it positively. 162 stated that the application of 
the computer-based learning environment was useful. On the given scale from 
1 (strongly disagree) to 5 (strongly agree), the arithmetic mean (AM) of the results 
was 3.7 with a standard deviation (SD) of 1.3. Since 3 would be neutral, the students 
evaluate the environment positively without being overwhelmed. 

After the course, the environment has been evaluated by the teaching staff. We 
conducted four interviews, three of them with student assistants who acted as tutors 
within the practical exercise and the exam preparation. They were responsible for 
adding contents to the knowledge map and for solving technical issues. One 
interview was conducted with the lecturer who was responsible for the overall 
coordination and who was involved in the planning and conception of the whole 
course. In the interviews, we asked the participants to rate several statements on a 
scale from | (strongly disagree) to 5 (strongly agree) and to explain their ratings. 
Moreover, we asked them to comment on the strengths and weaknesses of the 
environment and to suggest improvements. The students’ positive judgment of the 
environment has been corroborated by the teachers. For each statement, the arith- 
metic mean (AM) and the standard deviation is given (SD) (while interpreting these 
measures, one has to keep in mind that only four persons rated the statements): 


e The environment was useful for the students. AM: 4.25, SD: 0.43 

¢ The environment was useful for me in my role as a lecturer/tutor. AM: 4.00, SD: 
0.71 

¢ The students reached the learning goals better because of the environment. AM: 
4.00, SD: 0.71 
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e Ireached my teaching goals better because of the environment. AM: 3.50, SD: 
1.12 

¢ I would advise the students to use such environments more often if they had 
access to them. AM: 4.75, SD: 0.43 

¢ I would use such environments more often for teaching if I had access to them. 
AM: 4.67, SD: 0.47 

¢ I would use such environments more often for learning if I had access to them. 
AM: 3.25, SD: 1.79 (This is an interesting result: Why do the lecturers/tutors 
rather advise their students to use such an environment than use it themselves? 
The interviewees answered that their personal learning style is not optimally 
supported by such an environment, because firstly they prefer not to browse 
through learning contents but to study textbooks and other material, in particular 
exercises and exam questions from previous semesters, from beginning to end. 
Secondly, they prefer using pen and paper over doing all exercises with the 
computer. Therefore, they request an export to PDF so that they can print 
selected parts of the material.) 

¢ I consider the environment used within this course as a didactically sound 
means. AM: 4.50, SD: 0.50 


According to the interviewees, the strengths of the environment were, firstly, 
that the knowledge map gave a clear overview on the course contents and their 
inter-relationships. The students got a starting point for browsing through the 
material and exploring the themes independently. Questions could be answered 
by pointing to specific objects on the knowledge map, and students could (and did) 
answer their follow-up questions themselves by exploring the surrounding/linked 
objects. Thereby, the autonomy of the student was effectively supported. Secondly, 
the chat widget allowed fast feedback from the students. Questions could be 
answered immediately. Since all students could read the answers, questions did 
not have to be answered twice. Thereby, the tutors’ explanations became more 
efficient. The tutors saved time for helping with truly individual problems. Thirdly, 
the environment improved the communication among the students and, thereby, the 
collaborative learning. After a short time span, the students began to answer 
questions asked by other students. Fourthly, the environment rendered the students 
more flexible regarding their time management and learning speed. They were able 
to repeat lessons and exercises without losing track of the course or thwarting 
others. 

Concerning weaknesses, the interviewees mentioned technical and usability 
issues; in particular regarding the administration of the environment and the adding 
of new contents to the knowledge map. These issues have to be solved, but they do 
neither affect the concept nor the general design of the environment. Moreover, the 
interviewees propose the following extensions of the environment: 

The chat widget should be exchanged or supplemented by a forum for general 
questions and by a commentary function for the elements of the knowledge map. This 
would improve the linking of contents with questions and comments. They consider a 
learning planner consisting of a simple to-do list with links to exam-related material 
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and topics, self-tests and a visualization of the current level of knowledge/exam 
preparation progress (related to the self-test results) as extremely useful. The inter- 
viewees agree that the contents are the most important feature of the environment. 
These have to be updated regularly. So far, the contents of the knowledge map are 
explored by browsing. An additional search engine for the direct search of specific 
content would be reasonable. One interviewee deems a recommender system that 
recommends related external material useful. 

One aim of offering the ROLE environment was to support SRL. Has this goal 
been reached, that is, did the environment effectively support self-regulation? The 
interviewees claim that this is in fact the case. While in the beginning, a lot of trivial 
questions were asked, the students were able to find the answers to such simple 
questions themselves soon. (The question is, however, whether we can attribute this 
development to an improvement of self-regulation or rather to a learning effect 
regarding the course contents.) The interviewees considered it important to support 
SRL. They estimated that by far, most of their students had medium SRL-level. 
They correlated the SRL-level with the general knowledge level and acknowledged 
that students with a high SRL-level learned better and faster. However, as tutors and 
lecturers they generally preferred to teach students with a medium SRL-level over 
students with a high SRL-level. They justified this preference as follows: A tutor 
was supposed to lead interesting discussions with high SRL-level students. How- 
ever, they did not need a tutor that much and therefore did not get in close contact 
with them. Often, teaching did not really take place. Moreover, these students 
tended to be good students that asked difficult questions. A teacher had to be 
well-prepared and feel certain on the course topic to cope with these questions. 
This made it sometimes harder to teach students with a higher SRL-level. 

Medium SRL-level students were intelligent but still requested interaction with a 
teacher. The teacher got in contact with them, observed the learning progress and 
saw the positive effect of explanations and assistance. The interviewees found this 
very rewarding. 

The interviewees considered that a low SRL-level is correlated with rather low 
learning success. 

Teaching students with a general low level was considered to be cumbersome 
and not very rewarding. Feedback given through the environment was recognized 
by teachers as very important. The interviewees emphasized the role of the chat 
(or a forum). Feedback was deemed important for estimating the students’ progress 
and thus adjusting interventions. Moreover, it makes teaching more satisfying. 


Conclusions 


The evaluation proved the necessity of intensive promotion for new and additional 
e-learning tools. Tool objectives and advantages must be clearly communicated 
(at the right time) to the students. Nevertheless, only a minority of all students had 
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used the test bed for a longer time. Here, guidance with learning questions as in 
(Hsiao et al. 2010) may motivate students and foster communication. 

Until now, overview and learning guidance is given by the visualization of topic 
relations on the start page, the hierarchical and object-oriented organization of 
knowledge in the map and the linking of knowledge objects. The evaluation proved 
that the environment supports SRL and collaborative learning in large classes. The 
answering of student questions was easier via the chat widget than by email as all 
students were able to see the answer. Additionally, the chat fostered student-to- 
student support. Even if the test bed offered support for early learning, the peak of 
usage was reached just before the exam. It indicates the students’ remaining in 
power learning. 

The test bed was implemented as a cloud learning application combining 
widgets as services in an overall application and using IWC for communication 
between the widgets. Since different people were responsible for the particular 
widgets, it was sometimes hard to fix problems, e.g., when a server was not 
accessible. 

So far, the test bed was aimed to demonstrate the possibilities of ROLE techno- 
logy in large classes. The demonstration was successful and further development has 
to focus more on the learning requirements of students. Therefore, future improve- 
ments are seen in better communication and feedback support to strengthen, e.g., 
learning motivation. Suggested improvements comprise firstly a better collaboration 
support that can be implemented by adding improving, topic-related communication 
(forum, notepad linked to contents of knowledge map) and secondly a better 
SRL-support that can be implemented by adding a learning planner that supports 
planning (to-do list) and reflection (self-tests, visualization of progress). The offer- 
ing of learning strategies such as learning questions (Hsiao et al. 2010) within the 
learning tool may provide new advantages and motivation for the students. 


Lessons Learned 


The evaluation resulted in the following recommendations. These recommenda- 
tions are focused on the context of large classes in HE: 


1. New e-learning tools—especially if in concurrence to existing solutions— 
require intensive promotion. Students need a clear motivation and benefits for 
the own learning objectives. Multiple announcing will be helpful incl. situations 
when students will “hear” the message. Here, the usage of the exam preparation 
tool was fostered with an announcement only some weeks before the exam when 
student’s individual learning situations corresponds to the “message.” 

2. If a new e-learning tool must fit to an already existing process of learning and 
organization in HE, a good idea is better support for student’s individual learning 
process, e.g., when the student is not present at the university and not in contact 
with tutors. 
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3. During their learning workflow, students are interested to ask and answer 
questions. Therefore, e-learning tools for individual exam preparation are 
more attractive if combined with communication services as chats and email. 
Peer-to-peer communication within the students is welcome and can reduce the 
effort for mentoring by tutors. Even, communication can provide qualitative 
feedback for learning content as exercises and solution samples. 

4. One chat for all—instead of multiple chats for different content—provides more 
communication activity. Chat activities can link to the corresponding content to 
clarify the context for questions. 


Outlook 


The design of the complete course has been analyzed after the exam in September 
2012. The usage of the ROLE test bed, the participation and other indicators has 
shown that the students’ SRL capabilities do not meet the presumptions for the lab 
design. On the one hand, the students’ pre-skills in programming differ signifi- 
cantly; as a consequence, the strict schedule of the lab was often too slow or too fast 
for them. On the other hand, the capabilities for self-organizing teaching material 
from different sources are not very well developed and students expect to have all 
pieces of information at one place. The lab learning activities were often frustrating 
for the students and tutors. 

Therefore, we developed completely new teaching material which guides the 
students through the learning process. In the beginning, the basics of programming 
are explained in every detail and step by step, the autonomous work phases of the 
students get longer and longer. This teaching material supports an individual 
schedule of learning activity, as well, which corresponds better to different pro- 
gramming skills. First evaluation results show that the students are much more 
interested in the learning activities. Even the tutors prefer the new learning design 
since the students are motivated and ask interesting questions. 


Shanghai Jiao Tong University: ROLE for Employed, 
Part-Time Students 


The SJTU case study set out to address the issue of using Personal Learning 
Environments in adult higher education. It is associated with the SOCE whose 
blended classrooms are based on the Standard Natural Classroom model (Shen 
et al. 2008) providing face-to-face interaction with the instructor as well as online 
courses. Its students, mostly adult learners who have a job, take classes in the 
evening or at the weekend, by either attending in person in the classroom or by 
watching live over the Web. Improving their competencies via degree education or 
certificate training enables them to increase their chances in the highly competitive 
Chinese job market—a market that is also characterized by frequent job-hopping. 
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Teaching and learning in this case study follows a traditional pattern: it has a 
teacher-centric focus, with a “broadcast” model, where most students watch the 
lectures rather passively. The ROLE project, in this instance, has offered SJTU an 
opportunity to explore and investigate how to use existing available ROLE tech- 
nologies and tools that could provide a larger amount of opportunities for learner 
and teacher interaction enabling further potential creative ideas for both parties too. 
For instance, selected bespoke ROLE tools, such as those offering Voice Recording 
and Text-to-Speech recognition, allow foreign language students to practice their 
pronunciation by recording themselves and comparing their speech to the “original” 
one, thus providing the students with an active learning opportunity. 


Learning Scenario 


One central aspect of PLEs is that learners can assemble their own learning 
environments from existing services. They decide which services to use, assembled 
them, and use it for learning. Such a usage presupposes active, technical-savvy 
students. From our experience we knew that the students at SOCE do not fall into 
these categories. Most students have limited knowledge about Web tools (RSS is 
virtually unknown), only limited time at their disposal, and limited technical 
expertise. Furthermore, in the Confucian culture of China learning is still very 
teacher-centered (Zhang 2007), and students are not used to actively contributing to 
class. We therefore decided to build a PLE according to the learning scenarios 
specified by the teachers of the courses and make the pre-build PLE accessible to 
the students. 

One example scenario devised by the teacher of the French course is as follows. 
His course aims at helping the students mastering the first steps in spoken and 
written French as well as learning about and mastering tools that help students in 
their working life. These two goals are supported by activities that require using the 
tools. For instance, starting with an English (or Chinese) sentence, such as “Hello, 
my name is Tianxiang,” the teacher shows how to use a translation tool to get a first 
rough French translation, and how it can be refined by using a dictionary and spell 
checker. The French sentence is read aloud by a text-to-speech tool and repeated by 
the student until it can finally be recorded. This recorded introduction can then be 
uploaded to social networks. At a later point in the lecture, the students will receive 
a similar task, such as describing their job without being shown how to use the tools. 

Figure 5 contains a screenshot of a PLE, whose basic functionality is similar to 
the start pages Netvibes and iGoogle. It provides a single page from which the 
students can access different language related services and sites. The widgets of this 
PLE facilitate learning a foreign language. For instance, the top right widget 
performs spell checking; the second one below enables the translation of texts 
using Google Translate; the third widget accesses a text-to-speech synthesizer; and 
the bottom left widget allows the student to record and playback his voice. 
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Fig. 5 Screenshot of a SOCE PLE 


As another example of a learning scenario, for the course on Data Structures, the 
teacher wanted to use a PLE that supports rehearsing for the exam. This PLE 
consisted of a large number of exercises, which trained different concepts in this 
specific domain, such as linked lists, sorting, etc. The PLE was not tightly inte- 
grated into the weekly teaching, but made accessible at the end of the semester, a 
few weeks before the exams. 

To summarize, the PLE usage serves different purposes: the students have a 
chance to acquire knowledge about existing tools that will be helpful even outside 
class. Communication in a foreign language becomes facilitated and empowered 
when supported by translations tools and text-to-speech. The former allow under- 
standing and producing content that learners not yet master due to insufficient 
vocabulary. The later enables the students to listen to new texts, copied from any 
source or written by themselves. Together with a recording device, they can 
compare their speech to the artificially produced one. Furthermore, the PLE pro- 
vides opportunities to train domain knowledge, in this case multiple-choice exer- 
cises that cover topics taught during the lecture (gender of verbs, prepositions, 
linked lists, sorting, etc.). By reusing existing services we were not required to build 
our own version of these tools. The free text-to-speech service offers an astonishing 
quality close to native speakers that would have been difficult to achieve on our 
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own. By assembling all the services in one page, access to these services is 
facilitated. 


The Personal Learning Environment 


During the ROLE project, SOCE moved from a self-developed, proprietary LMS to 
Moodle’ as the Online Learning Environment. Thus, in a first phase, ROLE 
technology was developed and evaluated in an additional system (Liferay), which 
offers widget support. Once the shift to Moodle began, the ROLE evaluations took 
part in ROLE Moodle extensions. This section provides additional information on 
the technical realization. 

Technical realization: The technical realization of the PLE used at SOCE 
underwent an early and an advanced phase. In the early phase, SJTU explored 
first usage of PLE in a technical environment that allowed only rudimentary PLE 
features, as the advanced features possible at a later time by technology developed 
in ROLE were not yet available. In the advanced phase, on which we focus in this 
chapter ROLE technology was developed and integrated into the SOCE learning 
environment. 

Starting in 2011, SOCE moved from its proprietary LMS to Moodle. Moodle is a 
popular LMS to manage courses that is the de-facto standard among many educa- 
tional institutions. It is a plugin-based PHP application that can be extended by 
installing additional modules. These modules have to be installed on a Moodle 
server by a system administrator. The Moodle view, as shown to students and 
teachers, consists of a main center area and a rather narrow left column (see Fig. 5 
for an example). The center area contains main course resources, such as a wiki 
page, a forum, a lesson, a quiz, etc. 

Moodle’s flexibility and adaptability is achieved via visual themes and server 
side plugins, thus an intervention of system administrators is required every time a 
change should be done. Teachers and students are not involved in the process of the 
customization. Teachers, for example, cannot add or remove plugins on their own. 
Differently from Moodle plugins, widgets are client-side applications that can be 
added to a system by skipping server side installation, which makes them easy 
to add. 

Our OpenSocial plugin for Moodle allows a simple and teacher/student-driven 
extension of Moodle’s functionality. Once the plugin is installed to Moodle, a 
teacher can add a “Widget space” to the course, specify a set of widgets for it, 
and choose whether 1, 2, or 3 column view should be used for widgets display 
(Fig. 6). The resulting outcome (as displayed to students) is the page with widgets 
shown in the iGoogle in similar fashion, where students can work with several 
widgets simultaneously (see Fig. 5). 


T https://moodle.org/ 
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Fig. 6 A teacher creates a space with widgets for a course 


From the implementation perspective, the plugin consists of two main parts 
(Bogdanov et al. 2013). The first part is an engine that renders OpenSocial apps on a 
page. This engine is Apache Shindig which represents a reference open-source 
implementation of the OpenSocial specification. The second part is a PHP module 
that is responsible for a configuration of a page with widgets, adding and removing 
them to/from the page and gluing Moodle with the Shindig engine. The OpenSocial 
API provides the standardized way to retrieve and exchange information between 
different Moodle installations and other social networks, which improves data 
portability and interoperability. More precisely, widgets can query Moodle for 
data via the Shindig engine: they can retrieve the currently logged in user, the 
current course, its participants as well as save and get arbitrary data. The privacy 
and security are managed via the Shindig engine and it is in the full control of 
university administrators. However, a widget installed within a course runs on 
behalf of the teacher who added it and can retrieve/update information that teachers 
can normally do in their courses. Thus, teachers are responsible for checking the 
trustfulness of a widget, before adding it into their environments. The ability to 
retrieve course information and its participants is achieved via OpenSocial Space 
extension that allows widgets to adapt to the specific context of the course (con- 
textual widgets). For example, a wiki widget can save data for a course and restrict 
access to only people engaged in this course. The same wiki widget will behave 
differently being added to another course: it will have a different wiki history and a 
different list of participants. 


98 H. Vieritz et al. 
Bundles 


Two bundles were created by SJTU/SOCE: Creating an audio self-presentation and 
the SRL bundle. Both are available in the ROLE widget store.* 

Creating an audio self-presentation: The bundle for creating an audio self- 
presentation in French includes four main widgets: a translator widget, a spell 
checker, a text-to speech engine and a recording widget, and some additional 
tools such as a CAM widget, a business dictionary and a conjugation tool. The 
four main widgets are used to create a self-presentation in French language, the 
additional widgets are to assist student in his learning activities and to collect usage 
data. This widget bundle is helpful in a language learning context and can be used to 
complete different tasks, such as learning vocabulary, improvement of pronuncia- 
tion, producing of texts and audio-files, etc. The precise functionality of the widgets 
is as follows. 

The Translator widget allows user translating terms or sentences entered. The 
Translator is linked with a Translator homepage and a Dictionary homepage, which 
can be called up by user. User, who is a beginner in French, may work with this tool 
to translate his self-presentation text from English or Chinese into French. 

The Spell Checker widget serves to fine tune the translation from the Translator 
widget or any other (self-produced) text. User may examine his self-presentation 
text with help of this tool and correct spelling mistakes occurred by translation. To 
check spelling, type or paste a text into a text entry field and click on a “Spell 
Check” button. 

The Text-to-Speech Engine allows listening to the pronunciation. The user may 
listen to the pronunciation of his self-presentation text to create or to check his own 
audio-file made up with the Recording widget. To check the pronunciation, enter a 
term or a text into a text entry field and click on a “Say it” button. Voice (male or 
female), language (also British and US pronunciation for English language) as well 
as additional effects can be selected. 

The Recording widget can be used to check user’s own pronunciation or to 
produce audio-files such as pronunciation samples, audio texts, or presentations. 
With this tool the user may record his self-presentation and compare it with the 
given pronunciation of the Text-to-Speech Engine. 

Some additional tools, such as a Business dictionary and a Conjugation Tool 
(to check a modification of a verb from its basic form), can be added to the bundle to 
assist user in his/her learning activities. 

SRL bundle: The students at the SOCE of Shanghai Jiao Tong University are 
young adult learners who typically have a job and family, and this only limited time 
at their disposal. Also, the students are average learners in the sense that they have 
little knowledge about how to learn, specifically only limited SRL skills. 

For that reason, SJTU together with Koblenz and supported by FIT and Uppsala 
devised a bundle for supporting SRL (see Fig. 7). The bundle consists of three 
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Learning activity recommender (for the homework) 


On this page you find the tools necessary for the two task of the homework (creating the slides and completing the survey) the “To-Leam 
list” and the “Activity Recommender” 

The tools were shown during the lecture on Saturday, 21st 

In case you require help how to use the tools, have a look at the slides (TODO) and this video 


Screencast_AR-TL wmv 


` 


To-Learn list | toggle Activity Recommender | toggle 
Lr E E en 


Order Task: Select a strategy to create a presentation 


Select Learning Strategy: 


Wh-questions strategy for creating presentations 


Activities: 


Fig. 7 A screenshot of the self-regulated learning bundle 


widgets: the To-Learn list, the Activity Recommender, and the Contextualized 
Attention Metadata monitor. It also contains a video illustrating the usage of the 
widgets. 

The Activity Recommender widget (bottom right in Fig. 7) gives hints about 
how to process a given task. In this bundle, the task consists of “how to create a 
presentation.” With the tool, users compile a learning plan consisting of learning 
activities. The tool shows the current task, matching learning strategies, a list of 
concrete learning steps, and additional information. 

The To-Learn list widget (bottom left in Fig. 7) allows to compile and to modify 
a learning plan. Users can add, rearrange, delete and rename recommended learning 
activities or add own activities. The Activity Recommender sends learning activ- 
ities to the To-Learn list. Students can keep them, or discard them, as they wish. 

The CAM monitor tracks how students use the widgets by storing the events 
send from the other widgets in a central database. This widget is not directly of use 
for the students in this case, but allows evaluating their usage. 

The case study done at SJTU that evaluated the bundle highlighted several points 
that are of utmost importance when using such a bundle in class. First, due to the 
novelty of the widgets, students will probably be unclear about how precisely they 
should use each tool on its own. It is thus recommended that extensive support is 
provided and that instructions are available that explain the tools. Secondly, the two 
main widgets in this bundle interact, which is not frequently observed as a behavior 
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in existing systems. This has to be explained in the tool itself. The instructions 
should cover the usage but also clearly explain the purpose of the widgets, since 
again the offered functionality is seldom encountered in existing software. 


Evaluation and Methodology 


The evaluations of PLE usage at SOCE started in 2009 and continued until the end 
of the ROLE project in 2013. Each semester, PLE technology was used in a number 
of courses, mostly language learning courses, namely several English courses 
(English Listening, English Speaking, Critical Reading), and 2 two-semester-long 
introductions to French and German, but also a Computer Science course (Data 
Structures). The number of students varied over the courses and semesters. In 
average, for the language learning about 200 students were inscribed to the courses, 
about 20-25 came to the lectures in person and about 50 (25%) took the final exams. 
These numbers are typical for the second language (which is deemed as rather 
unimportant by the students). About 1,200 students per semester took part in the 
Computer Science course, with a similar percentage as for the language learning 
courses of students participating in the final exams. In each course, an example PLE 
was assembled by the teacher, supported by a team of researchers. In the language 
learning courses, the PLE was introduced and used in class, and the students were 
expected and encouraged to use it outside class. In the Data Structure course, the 
PLE was created at the end of the semester and students were supposed to use it for 
preparing their exams. 


Results 


Results from the PUEU surveys: The survey used at SOCE is based on the 
Technology Acceptance Model (TAM, cf. Venkatesh and Bala 2008) and consists 
of a set of statements that measure the Perceived Usefulness and Ease of Use 
(PUEU) of PLEs. The PUEU questionnaire has therefore constituted the “core” 
part of the surveys used among the users (educators, learners, employees, trainers, 
etc.) of the ROLE test beds. Naturally, the rest of the questions included in these 
surveys have been targeting the specific context of each test bed. At SOCE, surveys 
were created for delivery in both English and Chinese. Students were informed that 
they should complete them once they had finished their course of study. This was 
entirely voluntary. For example the survey from the French and German course was 
only completed by 20 of the 150 students. This was despite the fact that the bespoke 
Moodle spaces were regularly used by students in class and enhanced by being 
actively demonstrated by the teacher. The survey results show that 65% of students 
indicated that they used the PLE, while 20% stated that they did not use it, with the 
remaining 15% reporting that they did not know what was meant by the term PLE. 
Similarly it was also noted that students felt that their knowledge of e-learning was 
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quite good (65%); with the remaining 35% recording their knowledge as high and 
better. 

Students reported an overall positive experience of using the PLE. Using a 
Likert-type rating scale with a range from 1 (strongly disagree) to 5 (strongly 
agree) all but two students either agreed or strongly agreed with the specified 
statements, such as “I find a PLE useful for my work,” “I accomplish my work 
more effectively with a PLE,” “I find the exercises provided in the PLE helpful for 
my learning,” “It is easy for me to use a PLE,” “It is easy for me to use a PLE,” and 
so on (see Appendix x for the detailed results of this PUEU survey). Interestingly, 
the two negatively formulated questions, namely “I find using a PLE frustrating” 
and “I find interacting with a PLE requires a lot of my mental effort” received less 
positive answers, namely 7 students agreed or strongly agreed, and 13 students were 
neutral or disagreed. 

Several questions in the PUEU survey inquired whether students would like to 
modify their PLE by adding new widgets or by replacing widgets. The majority of 
students (15 or more) agreed with this premise. Furthermore, the perceived value of 
specific widgets was also investigated. They remarked that they thought that the 
preassembled PLE/Moodle space was well designed. This manifested as very 
positive ratings for all widgets. The actual Moodle usage logs, however, indicate 
that only a few ROLE widgets were actually used by the students (mainly the 
translation tool and some of the exercises), although the teacher encouraged usage 
in every class. The results from the survey, therefore, can be seen as representative 
of potential interest rather than reflecting active ROLE widget usage. 

Significantly the English students completed fewer surveys than those on other 
courses, and correspondingly their rating was also less positive. In the English 
courses it appears that the focus was on the exercises and, in contrast to the German/ 
French courses, where the English teacher used the PLE/Moodle spaces less 
frequently in class. The survey also revealed that the majority of English students 
did “not know what was meant by PLE” (more than half of the given answers). 
Those English students who declared an understanding of the term PLE also rated 
the perceived usefulness and ease of use positively albeit less than those students in 
the German/French courses. It also appears that the English students also are less 
inclined to assemble or change widget spaces (as evidenced by the answers to the 
respective questions). In addition, the responses relating to the perceived value of 
the provided tools appear to be mostly average, with only the translator and 
exercises being rated slightly more positive. 

Results from the Activity Recommender and To-Learn task list: In addition SJTU 
also conducted a further evaluation of two specific widgets, namely the To-Learn 
list and the Activity Recommender, within the Business English class. The 
To-Learn list enabled students to define and work on to-do-lists specific for 
learning. The Activity Recommender supported students in preparing a presenta- 
tion, and can add tasks to the To-Learn list if the student agrees. For this evaluation, 
the students were given the mandatory homework of preparing a set of slides for a 
product presentation and also needed to use a PLE space that contained the two 
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widgets. Due to the usage of the widgets, and the completion of the survey being 
mandatory, a large number of students (240) completed the survey. 

Initially a preliminary evaluation took place a few months before the deploy- 
ment of the second survey evaluating the usage of the implemented ROLE tools. In 
this first evaluation, the activity recommender was implemented as a mock-up using 
slides to introduce the widget. The immediate student feedback and a recording of 
the class’ reaction to the idea of this widget served to inform the ROLE partners of 
some potential shortcomings. The second evaluation, using the implemented tools, 
took place over a period of 4 weeks. The students received the (previously 
described) task to author a set of slides that portray a real or imagined product 
that a company would sell. The task was mandatory as homework. Students were 
also instructed to use the two ROLE tools during the authoring of their homework. 

The evaluation revealed several problems: those related to technology, e.g., 
features did not run due to browser issues and related to task/tool understanding, 
e.g., students did not understand the purpose of the tools. Thus, while the overall 
number of participation of the student was very high (given that this was a 
mandatory homework), the dissatisfaction of the teacher and students 
(as conveyed orally and by email) was also high, resulting in the suggestion that 
the tools needed to be improved for more effective usage in a classroom 
environment. 

The quantitative and qualitative outcomes arising from this evaluation are based 
on survey results with a sample size of N = 239. It can also be reported that the 
answers to the free text question “Did you have any problems using the Activity 
Recommender and the To-Learn-List?” revealed a series of technical issues as well 
as disclosing a lack of understanding among the student group that manifested as 
significant usability problems for them. That may have been influenced by the fact 
that SRL was a concept these learners were not used to. Since the level of English of 
the learners was relatively high, the problems were not primarily caused by 
comprehension problems. It is important, however, to take into account that the 
Learning Activity Recommender widget that they were using was designed origi- 
nally for a specific group of learners who would have needed substantial support 
and guidance related to their cognitive level and, therefore, was not designed with 
all learners in mind. Nonetheless the overall survey results can be interpreted as 
quite positive with regard to the perceived usefulness of the ROLE widget bundle. 
Since the To-Learn List in the evaluated widget bundle offered support for learners 
with good meta-cognitive competences it could be used a convenient tool. 

Widget Authoring Tool and teacher interviews: One of the major problems 
regarding the usage of ROLE technology at SJTU is that most teachers do not 
appear to be interested in many of them as they feel that there are too few 
appropriate widgets available that they can use in their courses. To overcome this 
problem, the ROLE staff at SJTU developed an easy to use authoring tool that 
requires little technical knowledge. In order that the authoring tool fulfills the 
teachers’ expectations and also meets the ease of use requirement, separate inter- 
views with five teachers were arranged in which a mock-up of the tool was 
presented to the teachers. During the interview, the mock-up was used to go through 
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the authoring process. This helped to identify any immediate difficulties and 
enabled the immediate collection of suggestions from the teachers. The tool has 
now been made available at SJTU. It uses ROLE technology such as inter-widget 
communication to capture interaction data and to allow students to rate widgets. 

The SJTU teachers involved in the courses using the ROLE technology were 
invited for a semi-structured interview with the aim to record their experiences with 
ROLE and how they would like to see ROLE improve in the final year. Two of the 
teachers accepted to be interviewed online and a third teacher accepted to provide a 
paper-based response to the interview questions. Overall the respondents thought 
that ROLE was extremely useful as it made possible to allow learners to access 
materials in a more flexible manner, enable them to self-assess their skills, enable 
them and teachers to have enhanced access to course metadata. 

All of the respondents expressed that they will continue to use ROLE tools in 
future because they were impressed by the benefits it brought to their learners (e.g., 
access to greater and better resources) and to them (e.g., monitoring, portability, 
etc.). The respondents zeroed on in two areas of improvement, which they men- 
tioned throughout the interview. Firstly the need to demonstrate the value of 
developing more contextual or subject specific widgets and bundles: 


I think what we have to do is to show more clearly the value that ROLE can bring to the 
teacher, so I think the basic technology is there. But it’s not every visible in the current 
widgets, the current tools that are available. I think this is one place where it really needs to 
improve. 

Maybe the developers should understand the course, because different courses need 
different learning pedagogies. The theory and experiment methods for different course are 
depended. 


Secondly, to make the widgets and associated technologies facing the end-users 
much easier to use without the need for any more technical knowledge than using 
Facebook for example. 

.... but maybe just something like Facebook, where I can just upload, share something, and 

press some simple buttons. And I think it’s still too complicated for me, and also for my 


students, as I described before. I’m getting a lot of questions. So I think the usability 
definitely needs to be improved. 


Conclusions 
Lessons Learned 


Two major lessons have been learned by the evaluations at the SOCE test bed. First, 
the significant role of guidance in such an HE setting, and second, the importance of 
having a sufficient amount of widgets. 

The first lesson became quickly obvious after the initial evaluations based on the 
Liferay system (cf. Ullrich et al. 2010). The initial approach of PLE usage consisted 
of introducing the PLE during class on the basis of an example. Then, the students 
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received a homework that required them to use the PLE as demonstrated. For 
instance, the teacher showed how to record a video using a Web 2.0 service, and 
the students’ homework consisted of recording a video, without a given specific 
topic. This open approach was motivated by the thought that the more open the task 
was, the more motivated the students would feel. However, this initial approach 
failed. None of students did this or any other similar open homework, interestingly 
although the students rated the PU of the PLE very high. In the next iteration when 
given more guidance by the teacher with specific tasks to perform the number of 
handed-in homework increased. We believe the low initial uptake despite PU is due 
to several reasons. First, students quickly become overtaxed. The concept of a PLE 
is unfamiliar, the embedded services are new to them, and they have only limited 
experience in Web 2.0 in general. Second, students often do not see the value in 
learning how to use these tools. They feel it distracts from learning grammar and 
vocabulary, and does not prepare them for the exam. Thirdly, most of the students 
(and teachers as well) are not intrinsically motivated to use Web services, and the 
majority of our students feel that the time could be spent more effectively. Thus, the 
task of the teacher becomes to demonstrate and highlight the advantages of a PLE, 
and guide them through it, so that the students can arrive at an understanding of 
what a PLE offers. 

Secondly, uptake of ROLE was significantly hindered since only few domain- 
specific widgets were available. In the case of SJTU, teachers were less interested in 
general purpose widgets, but asked for widgets covering very specific domain 
knowledge. Content available in existing Learning Object Repositories was not 
used in a single case, since these resources were too different from the specific 
needs of the teachers. For instance, existing learning objects about French were too 
much dependent of the original course book, and not usable in the SJTU courses due 
to too different vocabulary. Yet, teachers did find usable resources on Websites not 
available in learning object repositories. We therefore had to enable teachers of 
turning these Web resources into widgets usable in their PLEs by using a widget 
authoring tool. In addition ROLE staff at SJTU offer those courses that use the 
ROLE technology extensive support by technical teams who can set up the widget 
spaces and create widgets for those teachers who wish to avail of this service. We 
could observe that only those teachers who were extensively reported actually used 
ROLE technology. 

Finally, it is important to note that the discrepancy between the often very 
positive ratings given in the PUEU surveys and the often negative vocal feedback 
or confusion observed in classes related to ROLE technology, as well as the 
recorded low actual usage visible in the logs, indicate that, at least in a Chinese 
context at SJTU, the ensuing survey results should be interpreted with caution. 
Notwithstanding this word of caution, however, SJTU is still convinced that ROLE 
technology can enable the easy creation and usage of interactive activities that 
make the overall classroom activities more interesting and, therefore, empower 
teachers to offer extra learning activities that go beyond what a standard Moodle 
online course can offer. 
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Outlook 


SOCE will continue using ROLE technology after the official ROLE project has 
ended. The teachers who were using PLEs in their classes during the project’s 
runtime have taken over their PLE spaces into the course Moodle sites of the new 
semesters. Also, new teachers have expressed their interest in creating their own 
widgets during the presentations of the authoring tool. One teacher from the Social 
Science department created widgets of Web games about different political topics. 


Uppsala University: ROLE for Distance Students Working 
Collaboratively in Small Groups 


The Uppsala University test bed was performed within the distance course “Social 
software and Web 2.0” at Uppsala University. The course was given during the 
summer semester in June and July 2011 with 34 students and in the spring of 2012 
with around 20 students. The course corresponds to 5 weeks of full-time study. A 
university wide LMS installation was used throughout the course for the main 
interaction with the students. The course consists of four assignments each 
containing a part to be done individually and a part to be done within a group. 


Learning Scenario 


The test bed involved one of the four assignments in the course. The assignment in 
question aimed to give the students a deeper look into how social interactions are 
used in a professional manner, most specifically by a corporation that communi- 
cates with its customers via twitter. The students were tasked with finding patterns 
of behavior and how the social interactions work in a specific medium (in this case 
micro-blogging and customer relations). 

The goal of the assignment, as stated by the teacher of the course, was to use a 
typology presented by Shaw et al. (2011) to categorize tweets (i.e., 140-character 
statements made in the social media platform Twitter) sent to and from the Swedish 
train company SJ during the winter of 2010/2011. This particular winter was 
unusually problematic for Swedish train traffic, with extreme weather conditions 
resulting in severe delays all over the country. The assignment thus meant to 
analyze twitter discussions about traffic disruptions, mainly in commuting. Outside 
of the study, the students were required to read a paper coauthored by the teacher 
(Larsson et al. 2011) where a more summarizing typology was used to categorize 
the same tweets. The students were divided into six groups of 3—5 for the whole 
duration of the course, and were told to categorize the tweets collaboratively within 
these groups. 
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The assignment was carried out with the ROLE Uppsala prototype (a slight adap- 
tion of the ROLE SDK with single sign on for the students), mainly using a widget 
bundle that was developed specifically for the assignment. The bundle resulted 
from discussions with the course’s teacher about what tools he would find desirable 
for the assignment. 

Technical realization: The Analyze Tweets bundle” allows students to take a 
closer look at Twitter tweets. Figure 8 shows the bundle’s five widgets. First, tweets 
and tweet conversations are presented in a timeline. Second, the same tweets are 
presented in a list and can be tagged with 18 different categories. Third, a pie-chart 
of the amount of tweets tagged with each category is shown. Fourth, the tweets can 
be discussed in a forum and references inserted to individual tweets within posts as 
links. This allows students to, at a later time, refocus on the mentioned tweet by 
clicking on the link in the post (timeline and tweet list will adjust to show the tweet). 
Finally, any related content can be shown in the content viewer, for example, any 
links to Web pages mentioned in tweets show up here if clicked upon. This bundle 
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makes heavy use of inter-widget communications and most interactions performed 
inside an individual widget have consequences in other widgets. 

The bundle is intended to be useful when investigating tweets and the conver- 
sations they form. Students can make sense of the various ways people use tweets to 
communicate by using the provided vocabulary to categorize tweets. The forum 
allows students to discuss the problems that appear, for instance if some tweets do 
not fit into any of the provided categories. 

The purpose is to give students a chance of experimenting with one way of doing 
research in the area of social science. A teacher can assess students’ performance by 
looking into the categorizations that the students make as well as their activity in the 
forum. 


Evaluation and Methodology 


An initial pre-study was conducted as a survey during the summer course that gave 
input for the design of the full study in the spring of 2012. One of the results showed 
the importance of providing a prepared environment rather than relying on the 
students ability to assemble a suitable environment from scratch. On the positive 
side was that the students reported that multitasking and having multiple tools on 
the screen at once was nothing new to them (Jonsson 2012). 

Based on the pre-study, and the constraints on what we could do within the given 
course, a decision was made to test a limited set of functionality provided by ROLE. 
The choice was to focus on inter-widget communication, and how students perceive 
a user interface where multiple widgets could be used in combination to reach a 
predefined goal. 

The evaluation method chosen for the test bed was surveys since the context of 
the evaluation was a distance course at the university, where it was problematic to 
do interviews or use other methods where the investigator needs to be physically 
co-located with the users. The survey consisted of 28 questions and was answered 
by 16 of approximately 20 students taking the course (~80%). The reason for this 
approximation of the total number of students is due to a discrepancy between the 
number of students registered for the course and the number actually attending it. 

Out of the 28 questions 23 were formulated as statements and the students were 
asked to position themselves on a 5-grade Likert scale with the polar values labeled 
as “I fully agree” (1) and “I do not agree at all” (5). As we did not specify 
intermediate scale steps between the extremes, we can assume an interval scale 
between these. In order to detect survey artifacts some of the questions were 
formulated with a mirrored scale. Responses to these were inverted a posteriori to 
make it easier to interpret averages and correlations between statements. Of the five 
remaining questions four were free text and one consisted of multiple-choice 
checkboxes. 
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For more details about the evaluation, please see the extended report concerning 
the evaluation of the Uppsala University ROLE prototype by Lind and 
Laaksoharju (2012). 


Results 


Most of the students were at some point annoyed with something about the system 
(AM = 2.25, SD= 1.07). This judgment co-varied negatively with the impression 
that the system was working flawlessly. 

The students did not report any difficulties to learn how the system worked and 
generally considered the platform to supply good support in solving the assignment 
(AM = 3.50, SD= 0.82). That widget content was changing automatically when 
performing different actions in the system was conclusively seen as not confusing 
(AM=4.1, SD=0.93). They also seemed to think that system was relatively easy 
to work in (AM=3.19, SD=0.75). However, there is a tendency to judge the 
number of required mouse clicks to be too high (AM = 2.44, SD= 1.32). 

Students’ overview of their working process was perceived as good with an 
average of 3.5 and a standard deviation of 1.0. The fairly high standard deviation 
stems from responses being spread over the whole scale; though the majority 
(12 respondents) settled for a grading of 3 or 4, as reflected by the average value 
above and by the median value of 4. Significant correlations were observed between 
this statement and the level of participation and collaboration, the perception of the 
system as practical and the system’s impact on the motivation to complete the 
assignment. 

The students did not seem to have any significant problems understanding how 
to use the system for the assignment (AM = 3.75, SD = 0.86). Interestingly, this 
statement did not have any significant correlation with the other statements in the 
survey. 

The support from the system for doing the assignment was perceived as fairly 
high (AM =3.5, SD =0.82). This statement correlates with 11 of the other state- 
ments, making it the second best predictor after the perceived support for 
collaboration. 

Applicability in other areas was generally seen as high (AM = 3.38, SD = 1.03). 
Significant correlations were found between this statement and the perception of the 
system as motivating/tiring (inverted), the perceived support for collaboration and 
the statement that the student would have preferred another tool (inverted). 

Perceived support for collaboration received the highest count of correlations 
with other statements, twelve out of 22, making it a good predictor for overall user 
perception of system usefulness. The statement itself received an average of 2.75, 
with a standard deviation of 1.07, which is on the lower half of the scale, though still 
interpreted as relatively high considering the system is still in a development stage. 

The statement that the system was practical to use, received an average of 3.31 
with a standard deviation of 0.95. This indicates that the students’ perception of the 
system’s practicality was fairly high. This statement is also strongly correlated with 
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the perceived lack of problems in the system, the support for collaboration and the 
perception that the system increased the student’s motivation. 

The students felt to a high degree like they were part of a team while working in 
the system (AM = 3.94, SD=0.93). 


Conclusions 


Generally the results of the evaluation were positive. They indicate that the students 
were quite satisfied with the overall usability of the system and perceived that the 
system increased their motivation for learning and collaboration. 

Many of the students were, however, at some point annoyed with the system. We 
have seen in other studies that students have a fairly low tolerance to usability 
problems in systems that they are expected to use in their studies. The perception of 
the system as a whole may thus be biased toward a more negative impression than 
what would have been the case if the system setup had been more stable. 

An interesting, positive result when it came to the users’ perception of one of the 
unique features in this prototype—that widget content was changing automatically 
when performing different actions in the system— is that it was conclusively seen 
as not confusing; thus supporting this novel avenue toward automating internal 
communication between widgets. Initially there were some concerns that this 
technologically rather advanced approach would also be perceived as complicated 
by the users, a fear which proved to be unfounded. However, since the students did 
not get the opportunity to compose their own, unique set of widgets, we cannot 
determine whether this acceptance was due to a perception of the system as one 
united whole or whether the widget performance will be predictable even when 
users pick and choose widgets at will. Future evaluations should address this 
question. 

The results for how useful the different widgets were for solving the assignment 
are interesting, especially the low score for the tweet timeline. The ambition with 
this widget is to visualize patterns of interaction between Twitter users over time. 
However, either few of the students seem to have had use for this functionality or 
the interaction with the widget was not satisfactory. When looking at the classifi- 
cation data, almost half of the total classifications regard reactions and discussions, 
for which the widget should be a useful tool. The conclusion to draw from this is 
that the widget functionality was not apparent to the students, which can be due to 
either interaction problems or learning problems. 

The best predictor for the perceived value of the platform was how well it 
supported collaboration. This means that students who considered the platform to 
support collaboration also considered it to be valuable and vice versa. The conclu- 
sion we draw from this is that the collaborative aspects of the tool were something 
that the students expected, and either perceived as present or not. This suggests that 
the system was perceived primarily as a tool to support collaboration in the solving 
of an assignment. The fulfillment of the perceived purpose of a tool determines its 
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value assessment. This is also a good result for the prototype as it shows that the 
students who thought the system was a good platform for collaboration also thought 
that the system would be useful in the context of other courses. 

A perceived good overview seems to be very important for the overall impres- 
sion of the system as it correlated with other important features, viz., the level of 
participation and collaboration and the system’s impact on the motivation to 
complete the assignment. 

Generally the students felt highly engaged in the team effort of solving their 
assignment. It is not a controversial claim that a good overview in the system also 
positively affects the perceived presence of team members. Thus, if an important 
goal for the system is to stress the value in collaboration, creating a sense of good 
overview should be highly prioritized. 


Lessons Learned and Outlook 


The evaluation resulted in the following recommendations: 


1. Investigate why some participants perceived the platform as good for supporting 
collaboration and others not, as this appears to be an important determinant of 
the general impression of the platform. 

2. Keep the number of required interactions (like mouse clicks) at a minimal level. 
This can be achieved by exploiting the widget communication even more. The 
participants in the study did not have trouble understanding the automated 
behavior of widgets even though it conceptually appears rather complex. 

3. Reevaluate the platform in a course where it is possible to fully implement the 
SRL pedagogy. Currently we do not know how users of the system would cope if 
they were required to choose widgets by themselves. In the current study, the 
students might not be aware of the intricate, self-establishing nature of the 
widget communication and it is necessary to find out how this will be perceived 
when users set up the learning space by themselves. 

4. To address how complete SRL platforms are for students, future evaluation 
surveys should include questions about whether any other means of contacting 
group members were used in parallel, e.g., instant messaging (like MSN Mes- 
senger, Facebook chat), conference call (like Skype), collaborative documents 
(e.g., Google Drive). The aim should not be to create tools to replace existing, 
well-functioning communication solutions but to complement these. 


On the whole the results of the evaluation were encouraging and showed that 
continued work on the ROLE SDK and platform was motivated. 
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Widget Bundles for Formal Learning: Lessons Learned 


In this chapter, we described usage of ROLE technology in three different settings 
for formal learning in higher education. Despite the differences in the three test 
beds, the simple fact that ROLE technology was used to support teaching and 
learning at each of the locations supports the claim of flexibility made by the ROLE 
project. RWTH and SOCE were able to apply ROLE technology rather extensively 
and from early on during the ROLE project, and their evaluation results had 
significant influence on the later course of the project. The UU test bed took 
place at a later stage in the project and was therefore able to use a prototype very 
nearly resembling the project’s end product. 

The evaluation work done at these three test beds has taught the ROLE consor- 
tium valuable lessons that shaped the further work: 


e The added value of ROLE has to be clearly visible to its users. What became 
apparent in the two test beds SOCE and RWTH is that technology may not be 
attractive in itself, but its added value needs to be clearly visible to the users. 
This was voiced by users in the SJTU test bed, who have a full-time job, limited 
time available for study and low digital literacy, but also found in the RWTH test 
bed, which involves students with high digital literacy (similarly, evaluation 
outcomes from the BILD test bed pointed to the need for communicating the 
benefits of the ROLE project to organizations). 

e The value of ROLE that needs to be made visible includes technological 
solutions, but also psycho-pedagogical results, i.e., the benefits of SRL, group 
work, etc. These results and solutions have to be explained in a way the average 
user (organization, student and teacher) understands. 

e Uptake of ROLE technology significantly depends on the availability of widgets. 
If too few widgets are available for immediate use, then only few teachers will 
employ such technology. Having a few general purpose widgets, such as widgets 
teaching SRL or enabling discussions, is insufficient to attract users, instead they 
ask for very domain-specific resources. Such demand can be fulfilled by offering 
authoring tools. 

¢ Finally, the primary users of PLEs in formal learning in higher education in the 
three test beds described in this chapter were teachers, not students. In these 
settings, teachers used ROLE technology to integrate tools into the daily teach- 
ing that enabled additional learning activities. In such a setting, teachers served 
as multipliers, who demonstrate the potential of personal learning environments 
to their learners. For instance, the SOCE French teacher reported that having 
worked with the PLE during the semester for doing homework, his students 
asked whether and how they could access the PLE even after the lecture was 
over. They said they found it so useful; they want to continue using it. 
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Case Study 2: Designing PLE for Higher 
Education 


Denis Gillet and Na Li 


Abstract In this chapter, the concept of Personal Learning Environment is first 
refined taken into account recent advances and the experience gathered in a 
European research project dedicated to responsive open learning environments. A 
prototypal implementation of a Web 2.0 platform enabling the construction, the 
sharing, and the repurposing of personal learning environments is then introduced. 
Participatory design and validation activities carried out in the framework of higher 
education test beds aiming at understanding the benefits of personal learning 
environments in academic institutions are presented. Finally, the broaden applica- 
tion framework for the deployment and the adoption of open user-centric environ- 
ments for learning and knowledge management is tackled with perspectives in 
terms of supporting inquiry learning at school with online laboratories, 
implementing connectivist massive open online courses or enabling agile informa- 
tion systems for nongovernmental organizations. 


Keywords Personal Learning Environment (PLE) « Informal learning « Higher 
education * Social Learning * Web 2.0 


Introduction 


Trends 


Higher education is transforming under both top-down and bottom-up pressure. On 
one hand, national policies and international practices are pushing higher education 
institutions to reinforce their branding to attract more students to prime the 
researchers pump and to trigger more citations to boost rankings. On the other 
hand, students currently enrolled in higher education study programs are born with 
the Internet and have grown as teenagers with ubiquitous access to online resources 
(free or considered as such), peers, and communities, thanks to flagship search 
engines and social media platforms. A previous study (Vassileva 2008) has pointed 
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out that, today’s teenagers are used to learn in context, in response to a perceived 
demand, or to solve a particular problem. They search the Internet to find articles, 
videos, or any related materials, as well as scouting through their social networks to 
find a person who may be able to help. These learning experiences are mostly 
“solution-driven,” rather than “learning on principle.” 

As a consequence, the school- and course-centric closed learning management 
systems (LMS) do not fulfill and match anymore the institutions and students’ 
expectations and practices. They do not provide the necessary international visibil- 
ity for the institutions to be recognized as excellence teaching centers and they do 
not provide students with the open and persistent free access to resources and peers 
they are used to. The activity- and student-centric model of personal learning 
environments (PLE) better supports the opportunistic and agile scheme required 
by students to interact with resources, experts, and peers for both social and 
educational purposes. The LMS rather enforce the old teaching model focusing 
mainly on top-down content delivery through lectures, slides, course notes, and 
fully packed exercise sessions. 

The current trend is to equip institutions, teachers, and students with skills and 
technologies to combine /ocal institutional resources with global open content from 
the cloud, as well as to rely on internal and external support provided by peers or 
experts. 

Students have always exploited resources from libraries and relied on peer 
interaction for learning outside the formal institutional settings and activities. 
However, with the current ubiquitous access to information and communication 
technologies, such informal learning activities and modalities are taking a more 
important place in the higher education landscape (Gillet 2010). 

Activities carried out in the ROLE! project have contributed to investigate a 
pedagogical framework to strengthen self-directed and informal learning, as well as 
a technical framework to benefit from Web 2.0 PLE. One should also highlight that 
in the informal setting considered in this chapter, the boundary between learning 
and knowledge management (KM) is disappearing (Fig. 1). 

In the next paragraphs, we provide insights on the current definition of PLE and 
discuss alternative design and implementation approaches. Then, in section “Ded- 
icated PLE Platform,” we present the social media platform designed in the 
PALETTE (El Helou et al. 2009) and the ROLE (Gillet et al. 2010) projects to 
enable interaction in online learning communities and agile construction and 
exploitation of PLE by teachers and students. In section “Higher Education 
Test beds,” test beds set to validate the benefits of such platforms in higher 
education are presented together with evaluation results. In the ROLE project, 
these test beds also played an important role for the participatory design and the 
social requirement engineering processes. Finally, in section “Conclusions and 
Perspectives,” conclusions are drawn and perspectives are provided in terms of 
supporting inquiry learning at school with online laboratories, implementing 
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Fig. 1 Trends supporting 
the emergence of personal 
learning environments in 
higher education 


Higher Education 


connectivist massive open online courses (C(MOOCs) or enabling agile information 
systems for nongovernmental organizations (NGO). 


Definition of a PLE 


The concept of PLE is not new. According to Wikipedia,” it was first coined in the 
1970s. It was however rediscovered and consolidated with the emergence of the 
social Web (Web 2.0) that enabled users to take control of their online resources 
and interact freely with peers worldwide. Nowadays, conferences series such as the 
PLE Conference? are fully dedicated to discuss issues and investigations related 
to PLE. 

Initially, definitions of PLE were enforcing the combination of the online and 
physical resources to define the learning settings. As example, both the computer 
applications and the physical books the students may use on a desk at home to carry 
out a given activity can be considered as part of the corresponding personal learning 
environment. The same could be said for peers sitting around a table in a cafeteria 
and discussing or completing homework assignments. However, physical modali- 
ties and the associated offline actions are difficult to identify and to track within 
supporting online platforms; the tendency is then to ignore them or to consider them 
as embodied in the user model. Other definitions were mainly focusing on the 
concept of mashup of Web applications (apps), which happened to be too restrictive 
(Wild et al. 2008). 

The current conceptualization of a PLE is rather corresponding to an opportu- 
nistic aggregation of online content, information, services, and people for a given 


? http://en.wikipedia.org/wiki/History_of_personal_learning_environments 
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Fig. 2. The PLE as an 
aggregation of information, 
content, services, and 
people 


Information 


Services 


activity. The variety of physical interfaces enabling the access and the interaction 
with the mentioned entities is also evolving and is currently integrating not only 
desktop and laptop computers, but also mobile devices such as smart phones and 
tablets. Data gathered or published by online smart devices can also be considered 
as resources or channels. As a consequence, we can define a PLE as (Fig. 2) (Gillet 
2013) 


a shared online opportunistic and possibly ephemeral aggregation of communication 
channels, cloud resources, Web applications, and communities or peers (directly or through 
social media memberships), assembled in an agile way to define an interaction context for a 
given learning or knowledge management purpose, and accessed through interactive 
devices (computers, tablets, smart phones, ...). 


In this definition, communication channels correspond to live discussion streams 
or notifications usually made of short text messages or links. Content is referring to 
multimedia resources that exhibit some persistence. Loosely-coupled distributed 
services and online tools are accessed using Web applications. 

It is important to enforce that any digital ecosystem can be considered as a PLE 
if it has been repurposed for learning and knowledge management. It is more the 
intention of use than the design itself that defines the nature of the environment 
(Charlier et al. 2010). Obviously, the construction of the environment is part of the 
learning activities and is an important ingredient in the appropriation of the 
resources and the motivation to carry out the corresponding learning activity. As 
such, the PLE realm pushes further constructivism, by not only enforcing the 
definition of personal activities and resources by learners, but also by shaping the 
related interaction environments. As embedded in the PLE term, such environments 
are personal. They are however not individual as most of the time interaction with 
peers or experts is desired and supported. Finally, one should underline that each 
learning or knowledge management activity carried out by a user may rely on 
different tools and resources, and, as such may require the construction of a 
different PLE. This is the reason why PLE are often ephemeral, i.e., they may be 
abandoned once the activity is completed. It is also why their construction should be 
agile, i.e., they should be shaped to each context and purpose. 
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The above definition clearly shows the importance of the agile construction of 
the PLE by the users themselves. Such a construction requires quite high IT literacy 
and self-directed learning skills. The higher education framework is hence an 
interesting setting for validation of the PLE paradigm, as the acquisition of auton- 
omy is one of the main educational objective and outcome. Section “Conclusions 
and Perspectives” will especially show that students and early-stage researchers 
enrolled in graduate studies programs form an interesting community to elicit PLE 
requirements and evaluate PLE benefits. 

The ROLE project attempted to provide guidance for the design of self-directed 
activities and environments to students without the necessary level of autonomy, 
but had troubles to provide evidence that IT tools could help students with limited 
IT literacy to build IT environments on their own. For more autonomous students, 
the definition of personal learning objectives and the selection of the corresponding 
resources and services appeared to be generally carried out implicitly, requiring as 
such no specific technical support. 


Alternative Design and Implementation of PLE Platforms 


If one would stick on the PLE definition proposed in the previous paragraph, there 
should be neither design nor implementation of platforms enabling the construction 
of PLE. Any set of communication channels, cloud resources, Web apps, and 
people assembled by a user would become a PLE. As a matter of fact, most of 
the students in higher education build their own learning environments without 
identifying them as a PLE and even without noticing that they are actually building 
learning environments. A simple set of URLs, a LinkedIn* or a WhatsApp? group 
dedicated to a given topic studied could be considered as a PLE. 

As the concept of a free ecosystem assembled by a user fits very well with the 
definition of PLE, it makes it difficult to induce changes in the way higher education 
institutions shared and exploit knowledge and support students in both their formal 
and informal learning activities. It also makes difficult the development of IT 
literacy and autonomy for bachelor students in the usage of the social media 
platforms and channels for educational purposes. Social media platforms and 
channels are mainly considered as social interaction and entertainment tools. Last 
but not least, the current digital ecosystem solutions do not facilitate the storage, the 
sharing, and the repurposing of personal aggregations related to the study of given 
topics or the completion of specific activities. As a consequence, teachers and 
students have the tendency to continuously reinvent the wheel in putting together 
their learning environments. 


4 https://www.linkedin.com 
5 http://www.whatsapp.com 
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Following the above comments, it is still beneficial to adapt current LMS to 
enable a more flexible exploitation and enrichment of the learning resources and 
contexts by the students, as well as to provide alternative platforms to enable the 
construction and the sharing of PLE by the users for the users. The next section 
describes the Web 2.0 PLE platform designed and developed in the framework of 
successive technology enhanced learning research projects, especially in ROLE, to 
support the construction of PLE, their sharing and their repurposing. 


Dedicated PLE Platform 


Design Objectives and Models 


As stated before, a PLE is an aggregation of dedicated channels, resources, apps, 
and people by a user, i.e., typically a teacher or a learner, for a specific learning or 
knowledge management purpose. So, any platform supporting the construction of 
PLE should support the aggregation of the mentioned entities and their hosting. For 
the sake of symmetry in the way we treat the various entities being part of a PLE, 
we talk about the aggregation of people, which in fact means the ability of sharing 
the PLE with peers or experts and the possibility of repurposing it for their own or 
collaborative usage. When dealing with entities gathered from the cloud, due to 
their plethora, search and recommendation features are required. Finally, in an open 
framework where users may prefer different platforms to exploit their own PLE or 
the ones shared by others, import and export features in open Web standards are 
important (Fig. 3). 

The concept of PLE being abstract, for design and implementation purpose it 
should be materialized as an online context dedicated to support a selected indi- 
vidual or shared activity. As a consequence, we coined the concept of online spaces 
as 


PLE Platform 


Fig. 3 Required features 
for PLE platforms enabling 
the agile construction and 
exploitation of personal 
learning environments 
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the personal online places in which communication channels, cloud resources, Web apps 
and people are aggregated to support specific individual or shared activities. 


This space concept has first been introduced in Gillet et al. (2008), then formal- 
ized in El Helou et al. (2010) as the 3A model, and finally standardized in Bogdanov 
et al. (2011) as an OpenSocial® space specification. Hence, in the rest of this 
chapter, the notion of PLE, space and learning context are considered as equivalent. 
Obviously, a particular case of a share activity is an individual activity and a 
specific case of cloud resource is an internal resource (belonging to an institutional 
or an enterprise cloud). 

The agile creation by aggregation and the repurposing of a PLE being by essence 
a user-centric self-directed activity, the user should be able to set easily privacy 
settings and to assign specific roles and rights to the various people he or she is 
sharing the PLE with, i.e., the people he or she is carrying out the activity with. 
Such people are considered as the members of the shared online space. The lack of a 
fine control in the privacy settings and in the selection of people for specific 
activities is one of the reasons why LMS and flagship social media platforms are 
not very convenient to support the creation of PLE. The other reason seems that 
users do not want to mix their social and professional or educational networks 
online, and do not perceive a connection between their online activities and learning 
in classrooms (Greenhow and Robelia 2009). 

The notion of open educational resources (OER) is embedded and even extended 
in PLE. In effect, in addition to promoting the sharing of resources like multimedia 
documents, in the PLE framework we promote the sharing of Web apps offering 
dedicated services only required for specific activities and the sharing of the PLE 
themselves. In (Gillet and Bogdanov 2013), it is shown how a PLE instantiated as 
an OpenSocial space can be openly or privately shared as a meta-widget. 
OpenSocial apps (widgets, gadgets or meta-widgets) are in fact a combination of 
xml and javascripts. They can be compared to readable digital artifacts and shared 
using Creative Commons’ licenses. Such licenses are more suitable for the model of 
applications as a service (AaaS) relying on distributed infrastructures than the old 
open source licensing schemes working only for software that can be delivered on a 
memory stick as a standalone package. 


Prototypal Implementation 


Taking into account the specifications for a PLE platform stated in section “Ded- 
icated PLE Platform,” the Graasp* platform has been designed and implemented 
following an agile development methodology combined with participatory design 


© http://opensocial.org 
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and validation in test beds, including the ones described in section “Higher Educa- 
tion Test beds.” The elicited requirements could be summarized as the following 
design objectives for the platform that should: 


e Enable the creation of PLE without the intervention of system managers. 

¢ Be free of rigid structures and contexts (such as a course structure with specific 
types of activities and resources). 

e Enable opportunistic and focused activities. 

e Enable the aggregation of cloud resources. 

¢ Enable agile management of resources in contexts. 

e Enable the fine control of roles and rights in contexts. 

¢ Provide open search and meaningful recommendation of resources, apps, and 
people. 

e Rely on open Web standards similar to the ones used by social media and 
knowledge management platforms (avoiding carefully any specialized standard 
like the one dedicated only to learning). 


The core Graasp feature is to enable the creation and enforce the exploitation of 
dedicated online spaces as activity contexts (PLE). These spaces are defined, 
configured, shared, and populated by users, for themselves and for the audience 
they choose. Graasp stands for grasping resources, apps, activity spaces, and 
people. As a matter of fact, any space can embed subspaces supporting 
subactivities. However, hierarchy is not enforced. Users may decide to create either 
a flat or a hierarchical space structure to support their various learning or knowledge 
management activities. 

Graasp spaces can include members, resources, subspaces, and apps. In addi- 
tion, each entity has its own description implemented as a wiki enabling collabo- 
rative edition, a dedicated discussion thread, tags, and personal or public ratings, 
enforcing in such a way contextual exploitation (see Fig. 10 in Chap. 8). 

In Graasp, there are three audience levels. Spaces can be public, i.e., visible to 
everybody, closed, i.e., restricted to their members (but external people can request 
membership), or hidden, i.e., only accessible by invited members. There are also 
three possible roles for the members of a space. They can be owner, which means 
that they can add or remove resources, as well as invite members or revoke 
memberships. The owner role can be assigned to more than one member of a 
space, which is a unique Graasp feature that enables to pass responsibilities over 
when required. People can be contributors, which means that that can add resources 
and can create subspaces for which they have the full control of the participants. 
Finally, people can be viewers, which means they can access but not alter the 
content of a space. They can however post comments, which is an important part of 
the asynchronous interaction in shared online activities. 

The agile aggregation of cloud resources is supported by an open source plugin 
architecture and implemented as a GraaspIt! bookmarklet which provides a 
one-click aggregation of external resources in the Graasp clipboard for further 
integration in spaces (see the Chap. 8 for more details). 
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Offering a PLE platform to teachers and students in a higher education setting is a 
way to provide them with more opportunities and facilities to exploit and repurpose 
individually or collaboratively learning resources gathered from various sources. 
For adoption, such platforms should bring a high added value compared to institu- 
tional solutions extensively deployed like LMS. If a teacher wishes to share the 
slides of one lecture with students enrolled in a class, using the institutional LMS is 
obviously the way to go. However, if the objective is to help the students to develop 
teamwork skills by completing autonomously a collaborative project involving not 
only classmates, but also alumni or external experts, a PLE platform is a better and 
more agile solution. 

In this section, two test beds are described. First, the scenario and the evaluation 
of the exploitation of a PLE platform for teamwork activities as part of a bachelor 
course on Human Computer Interaction offered at Tongji University is presented. 
The design of social media platforms being part of the syllabus, the advantage of 
using a PLE platform in this case is twofold. It helps to illustrate the subject matter 
and it enables an easy management of the self-defined activities and the 
co-produced assets by the students themselves. Second, the support of inter- 
institutional training activities on Science 2.0 practices and solutions for doctoral 
students at the Swiss national level are detailed. Again, the benefits are twofold. The 
doctoral students discover how social media platforms can support their daily 
research activities and they also exploit the same platforms to exchange best 
networking and dissemination practices. 

These two cases emphasis the fact that using PLE or social media platforms in 
higher education is simultaneously a will to help users in developing IT literacy and 
teamwork skills, as well as a way to empower them in their learning practices. 


Teamwork at the Bachelor Level 


To examine the acceptability of Graasp in terms of supporting teamwork in higher 
education, it was used as a collaborative platform in the Human Computer Inter- 
action course offered at Tongji University in China during two consecutive years, 
i.e., in 2011 and 2012. Twenty-eight undergraduate students were enrolled in the 
course during the first year and 26 during the second. 

In addition to attending traditional lecture sessions, the students had to complete 
a collaborative design project using Graasp as a support platform. Both years, eight 
participatory design teams of 3—4 students were freely formed. The students spent 
about 10 h to design the mockup of a social media application. They also delivered 
a short report and gave a final presentation to their classmates and experts. 

After a short introduction to Graasp, students were encouraged to create their 
project spaces, share resources with each other, play different roles for the 
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participatory design (manager, designer, user, developer), and work with different 
project-oriented apps (task assignments, mockup edition, ...). A survey was 
conducted aiming at evaluating the acceptability of the platform in sustaining 
collaborative learning. Results of the 2011 evaluation have been detailed by Li 
et al. (2012). The importance of privacy settings for shared spaces was confirmed by 
the fact that in 2011 50 % of the spaces created were set to public, 47 % closed and 
3 % hidden, and in 2012, 60 % were set to public, 32 % closed and 8 % hidden. The 
students had also to pick five adjectives that closely matched their personal reac- 
tions to Graasp from a list of positive and negative words (Benedek and Miner 
2002). A word cloud, showing the frequency of the selected adjectives is presented 
in Fig. 4 and Fig. 5 for year 2011 and 2012, respectively. The word clouds show that 
the personal control, the ubiquitous accessibility and the trust enforcement of the 
PLE platform are part of the core added value. Results also show that improvements 
were achieved through participatory design between the two evaluations as effi- 
ciency and ease-of-use increased. 

Students’ answers show that they do not have traditionally support platforms for 
teamwork, for which they mainly use instant messaging applications (100 % in 
2011 and 59 % in 2012) or email (64 % in 2011 and 38 % in 2012) simultaneously 
or not. The introduction of Graasp was hence quite welcome. Students assessed that 
thanks to this platform they could accomplish their teamwork more effectively and 
that their motivation increased (only 10 % disagree with these statements). The fact 
that they can easily share resources with teammates, freely organized them, and 
seamlessly aggregate content from different sources were the main useful features 
according to the students (Fig. 6). 

This evaluation confirms that PLE platforms are useful for self-directed activi- 
ties carried out by bachelor students and involving agile resources aggregation, 
roles assignment as well as right management. 


accessible collaborative 


comprehensive connected Consistent 
desirable easy to use 
flexible fresh hard to use 
high quality. inviting organized 
personal relevant reliable 
slow time-consuming time-saving 
Fig.4 Awordcioudbased kgg technical trustworthy 


on frequency of selected 
adjectives in 2011 unconventionalusable useful Valuable 
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customizable easy-to-use 
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fun hard-to-use high- uality inviting 


organized persona relevant 
reliable slow time- 


saving trustwo rthy uncontrollable 


Fig. 5 A word cloud based on frequency of selected adjectives in 2012 


I can easily share interesting resources with 
my classmates. 


= Ican effectively manage my projects. 


I find it useful to work with different 
integrated Web apps. 


® Ican easily organize my resources using tags, 
sub-spaces, and links by drag and drop. 


= Ican easily aggregate resources from different 
sources into one place using Graasplt!. 


© Other reasons. 


Fig. 6 Usefulness of the Graasp platform for teamwork 


Science 2.0 Literacy in Doctoral Studies 


Despite the fact that more and more European universities are putting together 
doctoral courses or programs, most of the learning activities for Ph.D. students are 
self-directed. As a consequence, doctoral students are really eager to develop their 
skills to conduct research and learn from their peers. Acknowledging this need, the 
Federation of the French-speaking Swiss universities (CUSO”), which is offering 
inter-institutional continuing education programs, has decided to offer a soft skill 
workshop series co-organized by the University of Fribourg, the University of 
Geneva, and the EPFL to doctoral students of any discipline. 

Three of these 2-day workshops were organized in 2011 and 2012. The first one 
held in Geneva was described in (Bogdanov et al. 2012). The second and third one 
held in Fribourg and in Lausanne, respectively, are discussed here. 


3 http://www.cuso.ch 
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There were three objectives for the introduction of a PLE platform to support 
these workshops. First of all, it helped the educators belonging to the organizing 
institutions to collect and share their presentation material in advance and makes it 
available to the participants for preparation purpose prior the face-to-face sessions. 
This is a typical case when an open PLE platform is required as local institutional 
platforms cannot be used due to access restrictions for external people. Second, the 
PLE platform was exploited during the sessions by the participants to discuss and 
share their own resources and to practice their IT skills. Third, the participants were 
encouraged to continue to interact using the PLE platform after the events as an 
emerging community of practice. 

The main topics covered during the workshops were the search and exploitation 
of digital resources (especially scientific references), digital intellectual property 
rights, as well as Science 2.0 practices using PLE platforms and other Web 2.0 
tools. 

In the part dedicated to PLE, Graasp was quickly introduced. A 30-min activity 
was then organized during which the participants had to build a personal space to 
collect references and discuss the state-of-the-art of their Ph.D. thesis with peers 
and with their supervisor. The participants created the spaces they deemed appro- 
priate, invited selected people with the relevant roles, started to populate the spaces 
with chosen resources, and initiated discussions. This activity gave them the 
opportunity to discover the features of the Graasp PLE platform and experience 
with the benefits of contextual resource aggregation and discussion for a specific 
purpose. 

The evaluation carried out at the end of the workshops was quite general and 
covered all the topics and the platforms presented during the 2-day events. One of 
the clear and obvious findings of the evaluation was that the students were 
overloaded with new tools. Hence, these tools need to provide a very high added 
value and have the potential to be exploited for the all duration of their Ph.D. studies 
for convincing Ph.D. candidates to invest time to master and exploit them. The 
main added value that was elicited for Graasp was its ability to aggregate resources 
from different sources in dedicated spaces; the resources being either Web book- 
marks with previews or online documents. The other features were too numerous 
for the participants to really discover and benefit from them. Hence, we can 
conclude that in this case, the PLE platform was more useful for the educators to 
prepare, collect, and disseminate the teaching material in an agile and effective 
way, than for the participants. As a matter of fact, this is a general outcome of the 
ROLE project to highlight that sharing and repurposing of teaching materials 
among teachers, which are indeed self-directed learners, is an essential need 
which is effectively fulfilled with PLE platforms. 

A side outcome that was expected from this workshop series was that, taking into 
account that in Switzerland all doctoral candidates also act as teaching assistants, 
the participants would disseminate their best IT practices and preferred PLE 
platforms in their teaching activities with undergraduate and master students. 
Whether this goal was achieved has not been evaluated. 
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Conclusions and Perspectives 


The main purpose of PLE platforms is to bring more flexibility for organizations, 
teachers, and students in the way they aggregate and exploit online resources and 
services, while relying on peers and experts outside the formal class settings around 
which the traditional curricula are still organized. As a consequence, the application 
of the underlying PLE paradigms goes beyond the cases described in the previous 
sections. In fact, interesting new application domains have emerged recently and 
are described below. First of all, the concept of shared spaces integrating applica- 
tions happens to fulfill to need of inquiry-learning education at school using online 
labs. This case is described in section “Inquiry Learning Space for STEM Educa- 
tion at School Using Online Labs.” As the informal aggregation of resources from 
various sources and shared with various people is part of the definition of agile 
knowledge management, large NGOs are becoming interested in the PLE platform 
paradigm. The NGOs’ requirements and how they can contribute to the enrichment 
of PLE platforms are described in section “Supporting NGOs.” Last but not least, 
the current trend in delivering Massive Open Online Courses (MOOCs) using 
revamped LMS platforms brings back the old question of finding the right balance 
between teaching and learning in higher education. While the mainstream MOOCs 
platforms like Coursera or EdX emphasis paced video content delivery, the PLE 
platforms can enable connectivist MOOCs to be implemented by enabling teachers 
or students to assemble and control the exploitation of openly available learning 
resources. This case is discussed in section “Supporting Connectivist MOOCs.” 


Inquiry-Learning Space for STEM Education at School Using 
Online Labs 


The European Commission is funding for 4 years (2012-2016) a large-scale 
research project called Go-Lab aiming at promoting Science, Technology, Engi- 
neering, and Mathematics (STEM) education at school using online labs (Gillet 
et al. 2013). The solutions required to achieve this goal are trifold. First, Go-Lab has 
to strengthen and support communities of STEM teachers. Second, Go-Lab has to 
provide students with inquiry-learning spaces enabling the exploitation of online 
labs with proper scaffolds. Third, the online labs should be accessible through open 
Web apps facilitating their aggregation and repurposing by the teachers themselves. 

The ability of the PLE platform presented in section “Prototypal Implementa- 
tion” to support online communities, to create structured spaces, and to aggregate 
Web apps fully fulfills the Go-Lab requirements. In Go-Lab, Graasp is exploited by 
teachers to create inquiry-learning spaces in which they can organized all the 
resources necessary for the students to carry out the five typical phases of inquiry 
learning, i.e., orientation, conceptualization, investigation, conclusion, and discus- 
sion (Fig. 7). 
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Tags: O + wiry 


Description: Updated: Aug 12,2013 Added: Aug 12,2013 By: Adrian Holzer Viewed: 40 


Students will use the HYPATIA virtaul lab to determine the total momentum from all particles tracked after a particle collision and they will 
calculate (magnitude & direction) the missing momentum. 


Print descripton Emal description 


Members Spaces Resources A Apps 
Filter by name Sort by | default : 


| (Public) 
é= | 
| 
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Orientation Conceptualization Investigation Condusion Discussion 


Delete v Graasp view ~ Actions 


Fig. 7 The Graasp PLE platform exploited by teachers to create, share, and repurpose inquiry- 
learning spaces dedicated to STEM education (in this case to exploit the Hypatia tool provided by 
CERN to study to conservation of momentum in collisions of particles) 


The lower or higher secondary school students typically carrying out inquiry- 
learning activities having to concentrate on exploiting the online labs, obviously do 
not interact with the PLE platform. However, the export feature of Graasp enables 
the creation of a standalone version of the constructed PLE as an independent Web 
app. The teacher can create this app once the PLE is ready by clicking on the share 
button. A secret URL that can be shared with the students is then generated. It 
provides access to a simple Web page in which the inquiry-learning phases are 
represented by tabs under which the resources (including the labs) selected by the 
teachers and the instructions given in the embedded spaces’ wiki are accessible 
(Fig. 8). 

This example shows how a PLE platform can be exploited by teachers to 
construct advanced educational resources, which can not only be presented in a 
simple form to students, but also shared and repurposed by other teachers for their 
own classroom activities. 
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hypatia 


Description 
Students will use the HYPATIA virtaul lab to determine the total momentum from all particles tracked after a particle collision and they will calculate 
(magnitude & direction) the missing momentum 


Orientation Conceptualization Investigation Conclusion Discussion 


Here you can practice with the laboratory to find out how to operate it and what you can do with it. This will be useful to plan your experiment(s). 


Once you select a track from the table at the bottom you can see its track in the detector from two different points of view. The selected track 
appears in pink. In the table below you may see its momentun and its angle for both planes. 


Based on what you see in the detector, can you tell what kind of particles they are? 


p HYPA 


HYPATIA 1 HYPATIA 2 HYPATIA 3 HYPATIA 4 Exercise Instructions 


Fig.8 The simple Web page corresponding to the inquiry-learning space constructed by a teacher 
(Fig. 7) with embedded Web applications for inquiry learning at school 


Supporting NGOs 


The PLE and online contextual space features introduced in the previous sections 
have attracted the interest of global institutions, organizations, and enterprises, and 
especially nongovernmental organizations (NGOs), not for personal learning, but 
for agile knowledge management. These knowledge-oriented organizations are 
operating at a worldwide scale with distributed entities loosely integrated and 
relying on local hierarchical structures (Gillet and Bogdanov 2013). NGOs have 
also fast staff turnover and strong digital asset dependency, which require effective 
knowledge management solutions. The ability to create contextual spaces managed 
in an easy and agile way by freelance experts in the field or in operation is hence 
very attractive to such organizations. Their main additional requirements compared 
to the defined PLE platforms are: 


¢ Being able to integrate aggregation and interaction spaces in matrix structures 
(a space corresponding to a given project like Digging a Well in Ouagadougou 
should be simultaneously located in a geographical area (Africa as an example), 
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in an activity domain (like water management), and a hierarchical department 
(like operation). 

e Managing access rights by roles, i.e., the owner of a space should be identified as 
Project X Leader rather than John Smith (or both) to enable an easy transfer of 
duties and resources to someone else when required. 

¢ Enabling fine identity management such as fusion of user profiles when people 
contribute to spaces with various credential (such as a general gmail account or 
social media platform OpenID) for consolidation purpose. 

e Facilitating changes in access rights, i.e., having no distinction between the 
intranet and the extranet or the internal and external cloud infrastructures 
(as an example, when public documents are drafted and then made publically 
available, such feature eases the diffusion process). 

e Guarantying the secrecy of some resources as NGOs are often dealing with 
content which can be politically sensitive. As a consequence, the physical and 
geographical location of the cloud resources should be fully defined and 
controlled. 

¢« Automating tagging of contributed content to ease search and recommendation 
without diverting contributors from their main tasks. 

e Enabling proper licensing schemes (such as Creative Commons) for digital 
content so that public resources can only be used or shared under specific 
conditions defined by the owner. 


Such requirements are also very useful in the context of personal learning in 
which people often combine digital identities (e.g., when they move from one 
educational institution to another), where privacy issues are also critical (especially 
when dealing with young students), and where agile role management makes 
collaborative learning more effective. This section shows the interplay between 
personal learning and knowledge management in terms of supporting platforms and 
elicited requirements. In the future, design and validation between these two 
domains should be conducted more closely for cross-fertilization. 


Supporting Connectivist MOOCs 


PLE platforms have recently attracted interest from educators looking for agile 
solutions to develop connectivist MOOCs (Connectivist MOOCs 2013), referred as 
cMOOCs. Compare to the mainstream MOOC platforms like Coursera or EdX 
which are basically LMS open to external students, PLE platforms offer built-in 
social media features to boost opportunistic interaction and informal exchanges 
between students. These platforms can also help teachers and learners to aggregate 
their own MOOCs from resources freely available in the Cloud under Creative 
Commons licenses. 

The possibility to add Web apps in PLE platforms is enabling an easy adaption 
for their exploitation to support connectivist courses, and especially CMOOCs. Kop 
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et al. (Kop et al. 2011) highlighted that a connectivist course is based on four major 
types of activities, i.e., Aggregation, Remixing, Repurposing, and Sharing, which 
are the typical actions supported by PLE platforms. What is however missing in 
PLE platforms compare to MOOC platforms is the support for the formal activities. 
Especially, features to support coaching and assessment are missing, as well as 
features to support the timing and the structuring of the activities, including the 
associated content delivery and task assignment. Such features can easily be added 
in PLE platforms simply by integrating dedicated Web apps. This possible agile 
extension of the platforms through apps also enables the implementation of solu- 
tions to support a broad range of MOOC models, from the most formal to the fully 
connectivist ones. 

Requirement elicitation for CMOOC support in PLE platforms carry out with 
members of the RESCIF'° Network of Excellence in Engineering Sciences of the 
French-speaking countries have highlighted that the following features are espe- 
cially required: 


e Peer evaluation support. 

e Creation of quizzes, collection of the answers and analysis. 

¢ Team building and competence bartering support. 

e Formalization of time-based and topic-based structures through spaces (timing 
and navigation) using tables of content, syllabuses or calendars for navigation 
and exploration. 

e Support of additional metadata through internal tags (automatically identified or 
inherited from domain ontologies) to ease search and recommendation. 

e Customization of the portal spaces hosting cMOOCs with graphical templates 
enforcing branding or group identity building. 

e Management of multilingual resources (Wikipedia model) supporting a given 
activity to broaden the audience and the sharing opportunities with developing 
and emerging countries. 

¢ Tagging and subtitling of video sequences. 

e Online recording and editing of video sequences. 

e Integration of e-textbook standard documents (epub3). 


These features can however be provided as specialized Web apps. Once inte- 
grated in a space dedicated to a cMOOC, these apps are accessible to all members, 
can be personalized and can store or retrieve information related and resources 
belonging to this space. 

When relying on a PLE platform, a cMOOC implementation facilitates the 
co-production and co-exploitation of content between different teachers which 
can provide only materials directly related to their core expertise and rely on 
colleagues from other institutions for additional OER. Such an approach 
implemented using a mainstream platform would require challenging intellectual 
property right negotiations and bilateral conventions for exploitation. As such, the 


10 http://www. rescif.net/en/rescif 
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PLE platform not only enables to flip the classrooms (by freeing classroom time for 
personal interaction), but also to flip the institutions (by redefining the educational 
mission towards collaborative high-quality content edition and accreditation). 


Final Words 


PLE as considered in higher education correspond simultaneously to a paradigm 
change in the ways the information is shared and consumed, as well as a paradigm 
change in the supporting technical ecosystem. Recognizing and acknowledging the 
large variety of practices and platforms for both formal and informal activities 
carried out by teachers and students is already an important institutional change. 
Providing support in developing the necessary IT literacy of higher education 
stakeholders and even providing alternative PLE platforms helping them to increase 
their effectiveness in exploiting knowledge artifacts and exchanging competences 
are the next level of the ongoing academic revolution. Contributions to support this 
change have been formalized and illustrated in this chapter. 
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Case Study 3: Exploring Open Educational 
Resources for Informal Learning 


Alexander Mikroyannidis and Teresa Connolly 


Abstract This chapter explores the potential of informal learning within a Per- 
sonal Learning Environment (PLE), as well as the identified informal learning 
cultures that have evolved from the use of Open Educational Resources (OER). A 
variety of research instruments and strategies have been employed to promote the 
use of PLEs in this case study and capture a rich variety of feedback from 
Communities of Practice. In particular, there is a focus on the active use of a PLE 
and its integration with OER available from the OpenLearn project of the Open 
University. Additionally, this chapter describes the discovered necessary guidance 
conditions, emergent contrasting learning contexts and evolving different scenarios 
in use within the selected Communities of Practice. This research has led to the 
identification of valuable lessons as well as the documentation of challenges that 
are faced by those using PLEs in the context of informal learning scenarios. 


Keywords Informal learning ° Learning culture e Open Educational Resources ¢ Self- 
Regulated Learning * Personal Learning Environment 


Description of Case Study 


This case study focuses on the analysis of the informal learning opportunities 
presented by the Responsive Open Learning Environment (ROLE) project. Essen- 
tially, this research contains a series of different informal learning scenarios to 
examine, each of which will be assessed separately. The premise of informal 
learning (The Organisation for Economic Co-operation and Development 
(OECD) 2010) in this chapter relates to learning that has been gained through 
experience and not necessarily from an organised standpoint i.e. the opposite of 
formal education where pathways are often prescriptive and delivered from an 
instructor. 


A. Mikroyannidis (È<) * T. Connolly 
Knowledge Media Institute, The Open University, Milton Keynes MK7 6AA, UK 
e-mail: A.Mikroyannidis@open.ac.uk; T.Connolly@open.ac.uk 


© The Author(s) 2015 135 
S. Kroop et al. (eds.), Responsive Open Learning Environments, 
DOI 10.1007/978-3-3 19-02399-1_6 


136 A. Mikroyannidis and T. Connolly 


Additionally the information presented in this chapter relates primarily to the use of, 
as well as outputs of the OpenLearn project'—an Open Educational Resources (OER) 
repository of the Open University (OU). OpenLearn offers in excess of 12,000 h of self- 
study materials in a variety of formats. These include study materials repurposed as 
OER from original OU courses, i.e. initially designed for formal delivery, as well as 
new bespoke OER created by both OpenLearn academics and other non-OU educators; 
in other words, further facilitating OER available for informal delivery. As the majority 
of the study materials presented on the OpenLearn platform are made available using a 
Creative Commons licence,” it has been possible not only to reuse selected information 
but also to author new OER materials for the ROLE project, which can be freely shared 
to a worldwide audience. 

This chapter sets out to examine how an existing platform designed for global 
OER delivery, i.e. OpenLearn, can be enhanced with the introduction of Personal 
Learning Environment (PLE) technologies. It also considers a selection of social 
aspects of informal learning because this often plays a key role within a community 
of learners (Lane 2008a). These groups may have similar backgrounds or goals, so 
that a PLE can be used to support them throughout their informal learning process. 
With this in mind, it is also important to remember that the wider OU staff 
community is allied to the OpenLearn repository. Thus access to OpenLearn has 
enabled further investigations to take place in active Communities of Practice that 
contain a cross section of Higher Education (HE) staff: academic, academic-related, 
technician and librarian colleagues. 

A number of research instruments were deployed to gather data and information 
from these groups. This included using a variety of dissemination opportunities 
ranging from seminars relating to ROLE and the use of PLEs to the development of 
an interactive eBook describing Self Regulated Learning (SRL), plus presentations 
and conference publications. Table 1 outlines all the events and activities that were 
monitored, as well as their location and date. 

In this chapter, a selection of these events will be described in full detail. 
Evaluation activities will also be outlined and the subsequent analysis of the 
research instruments’ information presented in Chap. 3. Since many of the events 
and activities have overlapping learning contexts and objectives, they are illustrated 
initially in more general terms in the next section. Representative events will be 
described in more detail in subsequent sections. 


Learning Context and Objectives 


The first research instrument used in this case study is the seminar/workshop type of 
events. In terms of a learning context, these events comprised of an introductory 
presentation about the ROLE project, followed by the setting of a framework for the 


1 http://www.open.edu/openlearn. 
2 http://creativecommons.org/licenses/by-nc-sa/3.0/. 
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Table 1 Overview of the events, activities and artifacts that constitute this case study’s findings 


Case study Date Location 

1. Events (seminars and workshops) 

JTEL Summer School workshop May 2011 Crete, Greece 

SCORE Seminar July 2011 Milton Keynes, UK 

OU eLC Seminar January 2012 Milton Keynes, UK 

Build a Widget Day March 2012 Milton Keynes, UK 

JTEL Summer School workshop May 2012 Estoril, Portugal 

PLE Conference 2012 workshop July 2012 Aveiro, Portugal 

Dev8eD conference workshop May 2012 Birmingham, UK 

ITCM seminar July 2012 Milton Keynes, UK 

International Workshop on Cloud November 2012 | Antigua, Guatemala 

Education Environments 

2. Activities 

ICALT 2011 paper July 2011 Athens, USA 

Chapter in Collaborative Learning March 2012 Hershey, USA 

book 

OER World Congress conference June 2012 Paris, France 

ICALT 2012 paper July 2012 Rome, Italy 

PLE Conference 2012 paper July 2012 Aveiro, Portugal 

AACE E-Learning Round October 2012 Montréal, Canada 

Table discussion 

AACE E-Learning paper October 2012 Montréal, Canada 

3. Artifacts 

ROLE online course Summer 2011 OpenLearn website: http://tinyurl. 
com/role-course 

ROLE online SRL course Spring 2012 OpenLearn website: http://tinyurl. 
com/role-srl-course 

ROLE SRL eBook December 2013 | Apple iBook Store: http://bit.ly/ 


self-regulated-learning 


subsequent workshop element of the seminar, i.e. the use of widgets inside a PLE 
for learning purposes. All participants had access to a laptop or tablet computer in 
order to engage with the provided online learning materials. 

The workshop element of the events then allowed participants to be introduced 
to a hands-on and guided use of a PLE. Participants used one of the two bespoke 
ROLE online courses, generally sharing the experiences in groups of two or more 
learners. Initially the presenter gave a short summary of workshop objectives but 
thereafter the individual groups explored the ROLE online course(s) for them- 
selves, by following the self-study structured learning materials. The presenter 
(s) remained nearby in the room and circulated amongst the participants, answering 
questions as required. In the final section of the ROLE online courses, there was an 
opportunity to evaluate the experience and all participants were, therefore, invited 
to complete a short questionnaire. 
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The second type of informal learning activity involved presenting and, therefore, 
disseminating information directly about the ROLE project at a variety of interna- 
tional conferences. This activity took the form of a presentation followed by a 
question and answer session. Again, this gave the opportunity for the presenter to 
illustrate particular aspects of the ROLE project, highlighting individual and 
relevant PLE elements, as well as enabling them to indicate the existence of the 
bespoke ROLE online learning courses in the form of self-study interactive OER 
materials. Attendees were encouraged to visit both ROLE online courses post- 
event, in order to learn more about the project and to complete the short question- 
naire related to their learning experience of using the bespoke OER materials 
(Mikroyannidis and Connolly 2013b). 

Dissemination about the ROLE online courses also took the form of using 
posters. These were used as a visual medium to promote and signpost relevant 
information related to both the PLE aspect and also the availability of the structured 
self-study OER materials. Posters were displayed at a number of international 
conferences and proved to be popular talking points for those attending the events 
and, in effect, an unanticipated informal learning opportunity for enquirers. The 
posters gave a visual representation of the ROLE project and provided an oppor- 
tunity for individuals to enquire and learn more about PLEs from the ROLE 
representative(s) who presented the poster. Further promotional materials in the 
form of bookmarks were also available, enabling visitors to “take away” some 
tangible and memorable signposts with further details about different aspects 
of PLEs. 

A third learning context for this case study is a group described as “artifacts” that 
have provided bespoke opportunities for informal learning. This group includes the 
creation of both ROLE online courses. As indicated previously, these practical 
courses were often embedded into events that were focused on raising awareness 
about the PLEs. They were also available online for anyone to study at their own 
time. A further artifact that provided an innovative informal learning platform has 
been the adaptation of the ROLE online courses into an eBook. Initially, the content 
of the first ROLE online course was taken and adapted for presentation as an eBook. 
As a pioneering development itself the ROLE online course had been through a 
number of iterations and changes, ultimately being published with somewhat 
different emphasis in terms of content. Likewise, the ROLE eBook followed a 
similar pattern of development. Whilst the learning objectives for the eBook 
remained relatively similar to those in the original ROLE online course, 
i.e. introductory, the actual learning content of the published ROLE eBook was 
somewhat different in form. This was because the SRL online course materials 
were merged with the original ROLE online course content. The resulting interac- 
tive eBook thus contained the selected contents from both courses. 

In summary, it can be said that the underlying curricula presented in each of the 
previously described themed informal learning events or activities has been, 
broadly speaking, very similar: i.e. an introduction to the ROLE project’s purpose 
and, in particular, a focus on the use of OER within a PLE to support informal 
learning. On occasion, specific topics were presented, as required, to the different 
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Communities of Practice too. In addition, information relating to any necessary 
guidance for the associated workshop events was given where relevant. Further- 
more, the specific development of the two ROLE online self-study courses also 
enabled either attending participants, or those directed to the courses at a later date, 
to study the materials at their own pace and, potentially, at a time or place of their 
choice. Finally, the repurposing of the online course contents as an eBook has 
enabled the dissemination of information about PLEs and by using associated OER 
materials has facilitated it to be widely distributed as well as used in many further, 
and unanticipated, informal learning contexts. 


Setup and Organisation of Learning Activities 


As previously described, the range of informal learning events offered by the OU 
has been divided into three broad dissemination categories: seminar and/or work- 
shop events, conference presentations and thirdly the development of bespoke 
learning activities that have been tailored to meet the identified learning needs 
and objectives of particular Communities of Practice. Consequently, the associated 
setup and organisation of the related learning activities was also often custom-made 
to meet the needs of an event’s anticipated or identified audience. It is reasonable to 
state, however, that most of the embedded learning activities were developed from 
an original basic master set and then adapted and/or repurposed for each of the 
individual contexts thus actively implementing one of the fundamental themes of 
OER that of reuse (Hilton et al. 2010). The underlying premise of adaptation 
described here having been to identify an appropriate ROLE widget, then create a 
structured activity around it and subsequently produce a set of achievable self-study 
tasks related to the topic. 

How that learning activity was embedded into the individual event or into the 
online environment varied according to a number of factors such as adhering to the 
adopted pedagogic model of the event; assessing the availability and suitability of 
technology for the learning activity, e.g. considering the appropriateness of the 
delivery platform, such as Moodle or the eBook. It was also necessary to reflect 
upon whether the learning activity could be evaluated at a later date using relevant 
and appropriate research instruments too. The mediation of the learning activity 
also had to be well thought out. The basic master set of learning activities had been 
originally developed for use as self-study materials, thus necessitating the addi- 
tional “presence” of structured and carefully crafted feedback for the user to read, 
understand and so allowing learning to take place (Conole et al. 2011). Interest- 
ingly, in the many face-to-face events it was possible to offer supplementary 
feedback and answer participants’ queries instantly, as required, thus providing 
another unanticipated blended form of delivery. Mentors, as such, were not pro- 
vided for any activities as such but, as previously described, there were several 
opportunities in the many seminars, workshops, conference presentations as well as 
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alongside poster displays for presenters to give supplementary guidance and pro- 
vide signposts to further details about the use of PLEs. 


Learning Culture 


The OER-based informal learning materials that were developed by the OU initially 
adhered to distance-learning pedagogic principles. This basic premise was later 
adapted to suit a blended model of delivery according to the circumstances of the 
individual events. The learning culture as such could, therefore, be described as 
promoting or adhering to one of self-study further described as attractive to those 
who have a high level of SRL (Zimmerman 1989). Additionally, in general terms of 
the types of people taking part in the events it can be reported that the majority of 
those participating, online as well as face-to-face, were staff from HE institutions, 
often the OU but not exclusively so. As previously described, participants were 
drawn from a number of different Communities of Practice, including academics, 
academic-related-staff, technicians as well as librarian colleagues. 


PLE Intervention 


The overall PLE intervention in this case study was essentially established from a 
basis of embedding structured learning activities that used selected ROLE widgets 
and then delivered them via a web interface. The chosen platform employed to 
display both online courses was the LabSpace* area of OpenLearn, which uses 
Moodle for the delivery of materials and associated educational technology 
(McAndrew et al. 2009). The PLE in this illustration, thus, involved a Moodle 
environment as well as the use of the selected ROLE widgets that were embedded in 
the online courses, enabling participants to access a whole array of further OER 
materials in their informal learning context. 

Table 2 lists the contents of the two ROLE online courses. The first course 
introduced the concepts and technologies of the project. It provided a combination 
of tools and services that enable learners to build their own PLE based on their 
needs and preferences. It also established a course template that was subsequently 
reused for the development of the second online course focusing on SRL. The 
template was based on an original OpenLearn study unit format that had been 
established in the LabSpace area of the website (Lane 2008b). Associated struc- 
tured learning activities, using widgets, encouraged participants to explore and 
discover further OER materials or tools to enhance their knowledge. 


3 http://labspace.open.ac.uk/. 
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Table 2 The ROLE and SRL online course tables of contents 


A. The ROLE online course 


1.0 Introduction 


B. The SRL online course 


1.0 Introduction 


1.1 Overview 


1.1 Overview 


1.2 Learning outcomes 


1.2 Learning outcomes 


1.3 Definitions 
1.4 About ROLE 


1.3 Definitions 
1.4 About ROLE 


2. Example ROLE widgets 2. Self-Regulated Learning 
2.1 Introduction 2.1 What is Self-Regulated Learning 
2.2 Social search widget: Binocs 2.2 A typical learner: Marcus 
Activity 1: Search for OER 
2.3 Bibliography search widget: ObjectSpot 2.3 Travel scenario 
Activity 2: Search for references 
2.4 Videoconferencing widget: FlashMeeting 2.4 Flora’s learning approach 
Activity 3: Search for FM replays 
2.5 Collaborative authoring widget: EtherPad 2.5 Tim’s SRL approach 
Activity 4: Use the EtherPad 
2.6 Different learning approaches 
2.7 ROLE and SRL 
Activity 1: Assess your SRL skills 
3. Building a PLE 3. An SRL scenario 


3.1 Introduction 
3.2 Using Google 


3.1 Amanda’s SRL journey 
3.2 Amanda sets her learning goals 


Activity 5: Create a Google account 


Activity 2: Setting your learning goals 


3.3 Adding a ROLE widget to iGoogle 


3.3 Amanda looks for learning tools 


Activity 6: Add the FM widget to iGoogle 


Activity 3: Looking for learning tools 


3.4 The ROLE widget store 


3.4 Amanda uses the learning tools 


3.5 The Google gadget directory 


3.5 Amanda reflects on the process 


Activity 4: Reflecting on your learning 


4. Conclusion and bibliography 


4. The PPIM 


4.1 PPIM overview 


Activity 5: Using a PPIM tool 


Evaluation questionnaire 


Conclusion and bibliography 


Evaluation questionnaire 


The second online course, focusing on SRL, followed a similar pattern in terms 
of layout. It also has sections, learning outcomes, and embedded activities. Both 
courses were released under a Creative Commons licence, as indicated earlier, thus 
enabling their contents to be not only used or studied in situ but also to be “taken 
away” and, potentially, repurposed elsewhere within the terms of the licence. In 
other words, whilst the intended purpose of the two courses was to raise awareness 
about specific aspects of the ROLE project by introducing selected widgets in a 
PLE to access further OER resources, the design and deployment of the course 
materials as OER also meant that they could be reused by others too. Again this 
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adheres to a fundamental premise of OER and that materials, tools and technology 
should be freely shared and accessible to a wide audience (Hilton et al. 2010). 


OpenLearn as a PLE 


The OpenLearn project has changed significantly since its launch in October 2006 
(Lane and Law 2012). It was originally designed as an OER repository, using two 
websites: LearningSpace and LabSpace. It set out to offer a full range of HE 
academic subject materials ranging from the arts and history to science and nature, 
at all study levels from access to postgraduate. Commencing with 900 h of study 
materials in LearningSpace and 900 h in LabSpace, the now enhanced OpenLearn 
website currently offers in excess of 1,200 study hours. A number of changes have 
taken place since the launch, the most notable being the significant increase of 
available types and styles of study materials. In 2010, the OpenLearn brand also 
expanded in size and content to incorporate a significant number of both audio and 
video materials from the former Open2.net website—a platform that had been 
developed to support joint OU-BBC programmes designed to encourage public 
engagement with materials related to a variety of HE subjects. It has been reported 
that there have been 24 million unique visitors and approximately 320,000 regis- 
tered OpenLearn visitors (Lane et al. 2013). 

Through offering the original OER study materials, and then further developing 
the OpenLearn website to incorporate the Open?2.net resources too, the OU has also 
endeavoured to add value to its Open Content by deploying leading edge Learning 
Management System (LMS) technologies for learner support. At the same time, 
using such an approach, it has also actively sought to encourage the creation of 
informal collaborative learning communities. Alongside these developments, the 
OU has also pursued the development of international research-based knowledge 
about modern pedagogies for HE (Sharples et al. 2012). These improvements have 
also presented the ROLE project with an excellent opportunity to gain access to a 
wide cross section of learners and educators who engage with OpenLearn OER 
materials. 

The ROLE project has embraced OpenLearn, both in terms of building on its 
experience of creating OER study materials, as well as enabling access to some of 
its constituent informal learning communities. By using the LabSpace area of 
OpenLearn for the development of the two ROLE online courses it has also been 
possible to not only introduce the idea of a PLE but also offer direct access to 
selected ROLE widgets that permit end-users to create their own PLE. Whilst 
materials for the two courses were hosted on OpenLearn as self-study OER units 
they can, of course as previously indicated, be used in face-to-face settings too. 
Again, this enhances and improves the possibility of accessing a further variety of 
audiences that may have an informal learning context. OpenLearn also provided 
ROLE with the potential access to an unanticipated large global audience. 
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This case study consisted of a number of events, activities and artifacts that enabled 
different approaches to be taken to explore the use of OER materials for informal 
learning. The overall aim of the research was to ascertain how PLEs could be 
implemented for different groups of educators in the HE sector. The first evaluation 
objective, therefore, was to determine which of the selected ROLE widgets were 
appropriate for the different audiences, as well as assess the impact of introducing 
the idea of PLEs to those who may not have been aware of these technologies. It 
was important to gauge whether people were receptive to the idea of PLEs in 
addition to trying to determine if individuals were prepared to adopt any of the 
widgets in their teaching delivery. 

The basic research instrument deployed in the different events was the Perceived 
Usefulness and Ease of Use (PUEU) Survey, which has been based on the Tech- 
nology Acceptance Model (TAM) (Venkatesh and Bala 2008; Venkatesh and Davis 
2000). The questions in this survey were used in all the test beds of the ROLE 
project and were tailored thereafter for each event according to the composition of 
the surveyed groups. Necessary changes to the original survey were, in reality, 
minimal. The PUEU survey was available online,* where all data and information 
was also collated. It is important to note that each of the ROLE test beds in fact used 
the PUEU survey thus allowing further analysis across the project to take place. As 
noted earlier, the PUEU survey was also included as an evaluation opportunity 
within both of the ROLE online courses. Again it contained the original questions 
slightly tailored for these informal Communities of Practice. 

Where appropriate, additional research instruments were used. For instance, 
during the workshop elements of an event, the ROLE presenters were able to 
offer help to individuals or groups as required but in doing so could also observe, 
first-hand, any pertinent issues that arose. Whilst this was not a systematic collec- 
tion of data, more so observational and unplanned, it did serve as an excellent 
opportunity to see how ROLE widgets, in particular, were received, understood and 
used. Observational notes were recorded. This view of participants’ engagement, or 
not, with a PLE also enabled the presenters to gauge the usability not only of the 
selected ROLE widgets but also participants’ interactivity with the two ROLE 
online courses. In some respects these observations could be described, therefore, 
as informal research instruments as previously indicated. 

The presenters also could observe any perceived impact that the ROLE widgets 
had on an individual’s informal learning. This was, of course, more difficult to 
ascertain as an observer but was often reinforced when participants, on occasion, 
requested help repeatedly thus indicating that they were having difficulty in under- 
standing some aspect of the course. Furthermore, observation could also be 
employed to ascertain the acceptance of the PLE by different HE groups. Again, 


4 http://fit-bsew. fit.fraunhofer.de/pub/bscw.cgi/39523090. 
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whilst this was not a systematic collection of data or information, it did serve to 
supplement the recorded responses in the PUEU surveys. 


Methodology, Evaluation and Participants 


There are three themes for this case study (see Table 1). Firstly, the seminar and 
workshop group events. Secondly, the dissemination activities that involved 
presenting information directly about the ROLE project at a variety of international 
conferences. Finally, the third group contained a selection of mediating artifacts 
that provided opportunities for informal learning. Selected examples from each of 
these three groups will now be described in terms of research methodology. 


Event 1: The eLC Seminar 

This was an opportunity to introduce the ROLE project to the e-Learning Commu- 
nity (eLC) of the OU and took place in January 2012. The eLC has more than 
300 members including those from both academic and related staff in the OU 
campus as well as potentially comprising of many Associate Lecturers of the OU 
who are based throughout the 4 Nations and 13 regions in the UK. The eLC offers a 
regular programme of workshops and seminars to OU staff, also available to invited 
visitors, and covering a wide range of innovative e-Learning-related educational 
technology projects. 

The ROLE seminar presented an opportunity firstly to describe and then, in the 
workshop element, encourage the attending 20 eLC members to use the online 
ROLE courses. As previously mentioned, the self-study units include introductory 
text about the ROLE project and have various structured activities that enable the 
learner to use a selection of ROLE widgets. The ROLE widgets that the participants 
were invited to use are shown in Fig. | and are the following: 


e Binocs: A widget used to search for OER in a number of Web 2.0 repositories, 
such as YouTube, SlideShare and Wikipedia. 

e ObjectSpot: A widget employed for bibliographic searching in popular biblio- 
graphic indexes, such as DBLP and Google Scholar. 

+ EtherPad: A widget used for synchronous collaborative authoring of a docu- 
ment, where participants shared the OER they found in the other two widgets, as 
well as their experiences from using the widgets. 


By using the online self-study ROLE courses in the workshop, the 20 participants 
were also able to gain further insights into how to build or adapt a PLE. This was 
achieved by completing the online courses, which included guided access to the 
ROLE Widget Store? where further relevant, learning and teaching widgets are 


5 http://www.role-widgetstore.eu. 
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Fig. 1 A selection of ROLE widgets for finding and sharing OER 


located. After the initial presentation defining an overview of the ROLE project, its 
aims and specifically outlining the objective of PLEs, eLC participants were invited 
to follow the structured activities in the ROLE online courses. During the structured 
activities, the ROLE presenters moved around the room offering help to 
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participants as required and attempting to aid them with their engagement with the 
ROLE widgets on request. On the whole, this meant fielding a wide variety of 
questions, as well as offering constructive advice about how PLEs functioned and 
where to find out more information relating to this developing area. The ROLE 
presenters did not actively seek to intervene in the participants learning processes 
and only responded when requested to do so by individuals. In effect, their presence 
in the workshop part of the eLC event offered a blended learning opportunity to 
participants and similarly acted as an informal research instrument to observe 
participant interactions first-hand. 


Event 2: “Build Your Own PLE” Workshop, JTEL Summer School 
This event was held in Estoril, Portugal during May 2012. It was established 
originally by the EU-funded ProLearn project and subsequently supported by the 
European Association for Technology-Enhanced Learning (EATEL) along with 
other EU networks of excellence. PhD students in the area of Technology-Enhanced 
Learning (TEL) from across Europe spend the JTEL Summer School week learning 
about the latest trends in TEL, and exchanging ideas about their Ph.D. projects. 
The “Build your own PLE” workshop was delivered during the JTEL event. 
Some 14 students attended the session and used a variety of ROLE widgets in order 
to find learning resources and start building their own PLE according to their 
research interests. Both the ROLE online courses were used in the workshop. 
Other ROLE workshops were also held during the JTEL summer school and 
these included a coding session using the ROLE SDK (a development service that 
focused on communication and collaboration), a widget design session, as well as a 
session concentrating on personalised support for SRL. Thus the “Build your own 
PLE” event was one of a family of sessions focusing on the ROLE project. Once 
again, the “Build your own PLE” workshop enabled participants to use the widgets 
previously described in Event 1. 


Event 3: “Build Your ROLE” Workshop, PLE Conference 

The third annual PLE conference took place simultaneously in Aveiro, Portugal and 
Melbourne, Australia in July 2012. Researchers, educators and practitioners in TEL 
and PLE were brought together for a lively exchange of ideas, practices and visions. 
A number of ROLE partners, including the OU, delivered a half-day workshop 
entitled “Build your Responsive Open Learning Environment”. Participants were 
introduced to the ROLE tools and learning methodologies and were encouraged to 
use these tools in order to build their own Responsive Open Learning Environment. 
Additionally, they were able to design their desired tools, according to their 
learning scenarios and requirements, as well as submit the results to a subsequent 
ROLE widget competition.’ Once again, there was a focus on the two ROLE online 
courses within the workshop, along with an opportunity to complete the PUEU 
survey. 


° http://projects.kmi.open.ac.uk/role/pleconf-workshop. 
7 http://www.role-project.eu/WidgetCompetition. 


Case Study 3: Exploring Open Educational Resources for Informal Learning 147 


A paper was also presented at the PLE conference, concerning some of the 
emergent lessons learned from the OpenLearn test bed of the ROLE project. The 
presenter described the use of widget-based PLEs by informal learners who sought 
and discovered new OER materials as a result of using the ROLE widgets. The 
presentation took the form of a “speed-dating” style i.e. not a ubiquitous 
PowerPoint. A recording of this presentation and the slides were made available 
after the conference too (Mikroyannidis and Connolly 2012a). 


Event 4: The Dev8eD Workshop 2012 
Dev8eD is organised by the Developer Community Supporting Innovation 
(DevSCI), a community of developers in the learning provider sector. Thus the 
Dev8eD event, held in Birmingham, UK in May 2012, was designed for developers, 
educational technologists and users working throughout education, who wanted to 
further the development of tools, widgets, apps and other resources for education. 
The ROLE workshop was attended by ten conference delegates. Participants 
were first introduced to the ROLE project through a brief presentation and then had 
the chance to use selected ROLE tools during organised group activities. The 
purpose of these activities was to enable participants to understand how a PLE 
can be used to support them in their everyday learning and research tasks. Addi- 
tionally, participants had the opportunity to build their own PLE according to their 
own learning and research activities. 


Event 5: International Workshop on Cloud Education Environments 

A workshop was hosted by Galileo University in November 2012 in Antigua, 
Guatemala. It focused on the exchange of the relevant trends and research results, 
as well as the presentation of practical experiences gained while developing and 
testing cloud education environments, both from a teaching and a learning perspec- 
tive. This workshop raised awareness about both the ROLE project and the function 
of PLEs in cloud-based environments. Once again, the two ROLE online courses 
were used initially to attract the new external stakeholders, as well as to underpin 
this workshop. The workshop was focused on a cloud education environment by 
examining how such informal OER materials and services can be distributed using 
a number of different publication channels. 


Activity 1: The ROLE Poster and Other Publicity Materials 
The Paris OER Declaration was formally adopted during the 2012 World OER 
Congress held at the UNESCO Headquarters in Paris in June 2012. Over 550 del- 
egates including representatives of governments, educators, NGOs, and interna- 
tional universities attended the Congress, which was organised in full partnership 
with the Commonwealth of Learning (COL) and supported by the William and 
Flora Hewlett Foundation (USA). 

A poster was displayed in the exhibition area, promoting the two ROLE online 
courses.® The poster gave a visual focal point to which congress participants could 


s http://news.kmi.open.ac.uk/11/18424. 
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attend and enquire about the ROLE project. It was an excellent opportunity to 
promote and disseminate information about ROLE to a wide range of international 
delegations. This event proved to be an excellent opportunity to promote both the 
project and PLE developments in a significant OER global gathering. In turn and, as 
recorded earlier, the poster also acted as a visual mediating artifact that enabled 
informal learning about the ROLE project to take place. 


Activity 2: World Conference on E-Learning Paper 

This is an international conference organised by the Association for the Advance- 
ment of Computing in Education (AACE) and co-sponsored by the International 
Journal on E-Learning. It was held in Montréal, Canada, during October 2012. The 
conference serves as a multidisciplinary forum for the exchange of information on 
research, development, and applications of all topics related to e-Learning in the 
corporate, government, healthcare, and HE sectors. 

A paper was presented describing a number of the lessons learned as well as the 
best practices that were observed from the findings of the ROLE OU test bed in 
summer 2012 (Mikroyannidis and Connolly 2012b). A round table discussion, 
involving eight people, also took place hosted by the OU. It set out to explore the 
challenges associated with supporting SRL in HE. 


Artifact 1: The ROLE SRL eBook 

As previously mentioned, a fundamental aspect of OER is the ability to share and 
potentially, therefore, try to encourage the reuse of the developed materials (Hilton 
et al. 2010). It can also be argued that in doing this, it is possible to reach out, 
disseminate and make contact with new and, possibly, unanticipated audiences. The 
two ROLE online courses, for example, had been designed with this in mind: reuse 
and sharing potentially using multiple formats and platforms. The materials that 
were presented in the LabSpace area of OpenLearn were designed using structured 
content and XML. This enabled them to be transferable to other platforms as 
OpenLearn offers numerous export facilities, for example: Moodle backup, 
SCORM, and IMS package. OpenLearn also allows its structured authoring docu- 
ments to be used as databases (Hirst 2012). 

During 2012-2013, the OU has been exploring as well as taking advantage of 
new and innovative ways to widen participation in its courses and associated 
informal learning tools (Connolly 2013; Lane and Law 2012). Whilst there has 
been a focus on OpenLearn as a vehicle to achieve this, further platforms including 
Apple’s iBook Store and YouTube have also been used as opportunities for 
informal learning. The advent of the eBook has offered a new opportunity to 
harness not only existing structured content but also include levels of interactivity 
previously restricted to the LMS platform (Moodle). As a distance teaching insti- 
tution, the OU has always endeavoured to extend the boundaries of publishing, as 
well as take advantage of educational technology to do so. Thus “rethinking” the 
publication of printed books as eBooks has offered the opportunity to not only alter 
models of production (i.e. from print to online to mobile) but also to “open out” and 
extend the fundamental and familiar idea of a book by creating new and exciting 
experiences for the readers. 
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With this in mind, the publication of the ROLE SRL eBook (Mikroyannidis 
et al. 2013) has taken place alongside a growing series of interactive concept 
publications produced by the OU.’ It has taken advantage of the HTMLS technol- 
ogy to produce an eBook that can include interactive ROLE widgets and other 
inline resources. As a consequence, the ROLE eBook provides an introduction to 
new learning technologies that empower the reader in terms of SRL and by 
providing access to information about as well as using PLEs. In effect, it is selected 
content from the previously published two ROLE online courses that makes this 
interactivity and raised awareness possible. A selection of learning tools has been 
included that will help an individual to build his/her own PLE and encourages 
him/her to become a self-regulated learner too. Readers have an opportunity to try 
these tools through a set of interactive learning activities included in this eBook. 


The Evaluation’s Participants 


As previously mentioned the primary research instrument for the majority of the 
described events and activities in this case study was the PUEU survey. Essentially, 
the survey was used to gain an understanding of how participants from different 
Communities of Practice have attended ROLE events and perceived the usefulness 
and ease of use for a number of selected and presented ROLE widgets in addition to 
capturing their experience of using the two ROLE online courses. As indicated 
earlier, a number of further observational notes have been collected. The latter will 
only be reported here to verify and support information collected from the PUEU 
survey. 

The objectives for the overall evaluation, therefore, focused on participants’ 
understanding of the concept of PLEs, their use of specific widgets and capturing 
knowledge of their opinions and interactions with bespoke ROLE products such as 
the eBook. In this respect, the PUEU survey has proven to be quite useful as it 
embraced all of these objectives. The survey was easily accessible to all workshop 
participants and users of the ROLE online courses. 

The numbers and profiles of the people who attended each of the OU-led events 
varied enormously according to location and timing. In summary, it can be reported 
that the groups ranged in size from 10 to 50. The majority of participants were staff 
from the HE sector although one group was PhD students. The attendees’ age range 
appeared to be between 21 and 50. In general terms most people who completed the 
survey had some experience of TEL although few had either practice or full 
understanding of the potential that a PLE could offer. It is reasonable to say that 
there was a fairly even spread in terms of gender across all events. Table 3 
summarises the profiles of the participants in each event or activity. 


3 http://projects.kmi.open.ac.uk/ib/. 
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Table 3 Brief profiles of those participating in selected events or activities 


Events 


JTEL Summer School workshop 2011 


Numbers 


Occupation 


PhD students 


SCORE seminar 


HE e-Learning teachers 


OU eLC seminar 2012 


HE e-Learning teachers 


Build a Widget Day 


e-Learning trainers, managers 


JTEL Summer School workshop 2012 14 Ph.D. students 

Dev8eD conference workshop 10 e-Learning developers 
ITCM seminar workshop 10 HE e-Learning practitioners 
Cloud Education Environments workshop 50 HE e-Learning practitioners 
The PLE Conference 2012 20 e-Learning practitioners 
Activities 

ICALT 2012 paper 20 e-Learning researchers 
AACE E-LEARN conference round table 8 e-Learning practitioners 


Evaluation Results 


The evaluation results will not be described for each of the completed events and 
activities, but reported for the first group of events only. As previously mentioned, 
the main research instrument to be deployed for this case study was the PUEU 
survey. The majority of results of this survey has been recorded for the events and 
will, therefore, be presented here. A number of the observations made by the ROLE 
presenters will also be included. In addition, comments and quotes recorded by the 
participants via the EtherPad widget will also be presented where appropriate. 
Table 4 summarises the themed groupings of this case study. 


The eLC Seminar 


The respondents to the PUEU survey appeared to have an even split of knowledge 
amongst them in relation to TEL. Conversely, however, a significant 88 % of those 
participants felt that they had “some” rather than a “good” knowledge of PLEs. In 
other words it appeared that the group as a whole was relatively new to the idea of 
a PLE. 

Observation of the interaction with the EtherPad widget revealed that some 
participants were hesitant to use this type of technology and required encourage- 
ment from their peers or more experienced colleagues (either those in the group or 
from either of the two ROLE representatives who were facilitating the workshop). 
The types of information recorded by the participants ranged from anxiety to 
amazement that such tools could enable individuals to learn collaboratively. 
There was also clear evidence, however, that a more experienced group member 
took advantage of the EtherPad widget to communicate with a colleague in another 
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Table 4 The groups used in this case study 


Case study groupings Carried out Surveys deployed 
1. Seminar and/or workshop 9 Yes 
2. Conference dissemination 7 No 
3. Bespoke activity 3 No 


part of the room about some mutual work unrelated to the subject in hand 
(i.e. ROLE and PLEs). It should be noted, nonetheless, that the same person used 
the widget firstly to give advice to his colleague regarding more effective use of the 
tool before moving onto the separate non-workshop subject. What this also 
revealed, of course, is that despite encouragement to explore ROLE widgets it 
was not a compulsory activity and at least one participant chose to continue his own 
non-ROLE/PLE work as well as participating in the workshop. 

It appeared that the EtherPad widget was used more constructively to exchange 
as well as record pertinent PLE-related information. For example, one participant 
detailed a blog address that they felt to be “an interesting take on PLEs. Just a tad 
off topic... (Note to self this is the wiki)’. Others noted comments relating to 
enquiries about how each ROLE widget functioned. These ranged from “how are 
keywords supposed to work” to “cannot get ObjectSpot to show on iPad2”. Indeed 
this idea of recording questions was taken somewhat further by one participant who 
remarked: “Why do we need two widgets for search?” 

The final question in the survey asked participants to record their feelings about 
their use of PLEs. There was a 77 % agreement that PLEs would be slightly useful 
for participants work, followed by a slight disagreement that the same PLEs would 
help participants accomplish their work more effectively than their current use of 
learning technology. Again this was not surprising as most group members were 
established and experienced users of learning technology and had revealed that they 
only had limited use of PLEs. Half the group proffered a neutral response to the 
statement relating to “It would be easy for me to use a PLE” whilst the remainder 
recorded that there was a slight chance that that would be the case. There was a 
more even spread of responses to the statement “It would be clear to me how to 
assemble a PLE using widgets” ranging from slight disagreement (11 %) through to 
slight agreement (also 11 %). Most participants remained neutral on the subject. 

Interestingly, the statement “I would find using a PLE frustrating” invited the 
most disagreement to be recorded with the majority (55 %) remaining neutral 
alongside 33 % saying they slightly disagreed and 11 % strongly disagreeing. 
Once again, the statement “I would find interacting with a PLE requires a lot of 
mental effort” statement invited a strong neutrality (55 %) yet 22 % of participants 
strongly disagreed with this premise whilst 11 % recorded that they slightly agreed 
that this would be the case for them. 

The remaining part of the survey related to participants’ motivation to using a 
PLE in their learning process whereby 55 % remained neutral in their responses and 
44 % slightly agreeing with this statement. The last statement of “I predict that I 
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would frequently use a PLE if I had access to it” invited an even response (33 %) 
between slightly disagreeing through neutral to slightly agreeing. 


The JTEL Summer School Workshops 


A ROLE workshop took place in both the 2011 and 2012 JTEL Summer Schools. 
The participating research students were enthusiastic and willing to try out the 
widgets within the ROLE online courses. Each workshop had the same format of an 
introduction to the ROLE project, followed by the practical activity of using the 
widgets within the ROLE online courses. In general terms the research students’ 
overall opinion in both workshops was a positive one. They engaged with the 
structured activities, actively used all of the provided tools as well as recording 
their thoughts (and sometimes frustrations) in the EtherPad widget. 

The EtherPad widget was used in many different ways in the 2011 workshop. 
Most of the research students used the tool although some were a little surprised by 
the real-time aspect of it: “. . somebody is writing on the screen!!!! I am scared. ..” 
Others considered additional aspects to the experience in that it highlighted some 
potential gaps in their own skill set: “...J probably have to work on my search 
Skills...” By contrast in the 2012 workshop, however, the EtherPad widget was 
used actively by only a few of the participants. Generally, it functioned as a means 
to record and exchange URLs of relevant resources such as the “Learn Portuguese 
language vocabulary” YouTube video'” that the participant described as: “This is a 
great video”. It was also used in identifying a Stephen Downes Slideshare presen- 
tation about “Personal Learning The Web 2.0 Way”! 

All participants in both workshops were aged between 20 and 40 years. There 
was also an equal 50 % male/female split. In answer to the question relating to the 
participants’ knowledge of TEL, in the first workshop there was a significantly 
higher response rate to the “some” option whilst in both workshops most stated that 
they had a “good” knowledge of TEL. In the second workshop with regard to the 
question related to PLEs, however, there was a greater spread of responses: 50 % 
recorded that they had a good knowledge whilst 25 % stated a “good” knowledge 
and the remaining 25 % claimed to be an expert in the field of PLEs. The free-text 
responses within the questionnaire provided some insight into the participants’ 
views of PLEs as well as the use of the ROLE widgets. 

The question: “What did you think of the widgets of the workshop activities?” 
also invoked a variety of responses in both workshops, most acknowledging that the 
widgets were interesting. In addition others said: “Z have found them very useful for 


!0Learn Portuguese language vocabulary YouTube video http://www.youtube.com/watch? 
v=bzR1q3ZAIKQ&feature=youtube_gdata_player. 

11 Stephen Downes Slideshare presentation http://www.slideshare.net/Downes/personal-learning- 
the-web-20-way. 
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my PhD and for my learning” along with a remark that (the widgets were): “Small 
apps which can expand your daily routine”. Others simply said that the widgets and 
interface: “look good’. It is also important to note a word of caution too though, 
summarised by one respondent who remarked: “Found them quite interesting. 
Collaborative text editor had lower quality in contrast to GDoc, surely I would 
not use it”. 

In relation to the question: “Were you able to find suitable widgets for building 
your PLE during the second workshop activity?” there was almost unanimous 
response in the second workshop alluding to the intermittent internet issues that 
appeared to plague the entire session. One quote, perhaps, sums up the frustration 
that most participants felt dominated, their experiences: “Internet connection issues 
did not let us perform this activity’. Nonetheless, 90 % of the same participants 
responded positively to the next question: “Did you find the workshop activities 
useful for your research needs and goals?” which can be summarised by one 
remark: “I found it an interesting approach to be tested in the future”. One 
respondent did, however, offer a rather more circumspect response: “J do not 
know. I have to check those pages more when I come home”. 

Opinions from both the workshops’ attendees about widget-based PLEs were, in 
general terms, evenly spread. One notable exception, however, was the response 
from workshop 2: “I would find interacting with a PLE requires a lot of my mental 
effort’. Nonetheless, the respondents overwhelmingly recorded that they were 
neutral in their feelings about this statement for the second workshop. Research 
students in the first workshop appeared to be more discerning in their learning and, 
as noted by the ROLE presenter, the majority of the students appeared to focus on 
the Binocs widget rather than the ObjectSpot widget. Again this seemed to colour 
their view of the overall experience of using the ROLE technology. 


The Dev8eD 2012 Workshop 


This workshop was attended by developers and learning technologists, predomi- 
nantly male and in the age range of 20-50. Most participants recorded that they had 
a good knowledge of PLEs but some also declared little or no knowledge of this 
area. Once again the EtherPad was used during the event to record notes and 
information relating to the workshop that participants wanted to share with each 
other. In this event, however, the participants took a more strategic view of the 
EtherPad by using it to store personal observations such as: 


Ive used Etherpad before. The problem with these synchronous writing tools is the way the 
connection suddenly stumbles and your flow is disturbed—Just had this problem with 
Etherpad. It was static for a while so I assumed people were still having connection 
problems—then I clicked to type and got a huge update! 
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In addition others used the EtherPad to record their thoughts about the activities 
themselves as well as how the widgets worked or performed, for example: 


Some searches return “60 results” apparently—all of them YouTube videos—I untick 
YouTube as an option and get 54 results—but there were way more than 6 YT videos in 
my previous list. 

The term ‘reflection’ isn’t especially useful for search... Binocs’ first result may be the 
kind of ‘reflection’ Im looking for but ObjectSpot results cover a range of different types of 
‘reflection’. 

At the end of the workshop, participants were encouraged to complete the PUEU 
survey. Again this gave them an opportunity to record their thoughts about both the 
ROLE widgets, the implementation of them in PLEs as well as the activities of the 
workshop itself. Responses for the question: “What did you think of the widgets of 
the workshop activities?” were mainly positive but with reservations about the 
mechanisms that were used to make the widgets function. Participants were con- 
structive in their observations saying, for example: 


Could be useful, though a few flaws here and there. Binocs had a odd way of searching and 
filtering. Etherpad is a great idea but it didn’t always sync correctly and would jump-start 
again when clicked. The Mash-Up Recommender is great but not all widgets were 
installable to iGoogle! 


There was also positive affirmation that the workshop enabled the participants to 
be introduced to new widgets, for instance: “The widget can be useful to put 
different tools together” and: “Useful because I had never heard of or seen these 
widgets”. Recognition, however, was also given to the technical issues such as: 
“Some problems with Etherpad on the iGoogle page—would be better if it sized 
down. Also the Binocs broke”. 

The question: “Were you able to find learning resources that relate to your topic 
(s) of interest during the first workshop activity?” also invited a variety of 
responses. Some participants: “Found some things but would need more time to 
explore—will do that soon” whilst others were circumspect: “yes, though would 
like to understand better why two separate search boxes. I'm guessing one is API 
driven, one is custom Google search? could they be combined?’. Remaining 
responses affirmed that the participants were, generally, happy using the ROLE 
widgets, for example: “Yes, I searched for 6lowpan and found some relevant videos 
and slides” and “I tried one topic relevant to my institution. The resources were 
good”. 

There was also overall positive response to the question: “Were you able to find 
suitable widgets for building your PLE during the second workshop activity?” once 
again, however, the participants did not hesitate to record their actual experiences 
of installing widgets or not, for example: “The ones I did find looked useful but 
didn’t add to iGoogle (there was an error)’. Another participant also remarked: “J 
think so, but I'd need to think about how Id integrate it with my other tools (like 
Evernote, Twitter, Google calendar)” demonstrating that as a developer or educa- 
tional technologist that they were giving some thought to the application as well as 
implementation of the ROLE widgets. 
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There was an overwhelming positive response to the question: “Did you find the 
workshop activities useful for your research/teaching/learning needs and goals?” 
once again summed up by: “Yes, a useful overview/primer of what’s possible” as 
well as: “Yes, Very useful”. Similarly the final question of the survey offered 
participants the opportunity to add any remaining questions, comments or sugges- 
tions that they wished to record. Some insightful comments were made such as: 


I still have lots to learn about this area but this was an informative session to get me 
started!” as well as: “I can’t help thinking that if I have the digital literacy skills and 
confidence to create a PLE then I don’t need a PLE. (a bit of a paradox!) 


Lessons Learned 


From earlier, interim, research about this case study (Mikroyannidis and Connolly 
2013a) three main themes were identified: 


¢ The usability of the learning tools, i.e. the widgets. 

¢ A consideration of the types and styles of related learning activity formats (often 
embedded in the ROLE online courses). 

¢ A reflection as well as action upon suitable methods that might encourage future 
participants to consider, engage and continue using PLEs for their own learning 
purposes. 


In respect of the additional events, activities and artifacts described here it can be 
reported that a number of further lessons have been learned. These will be consid- 
ered in the context of successes and failures. The underlying lesson learned was the 
importance of ensuring that all the technologies are stable, available and accessible 
at the time of engagement as this leads to successful deployment. Additionally, 
planning an event, activity or even an artifact should also include a level of 
adaptation or localisation for particular audiences e.g. PhD students have different 
requirements to the more experienced researchers. 

The creation of the first ROLE online course alongside the development of the 
second ROLE online course focused on SRL were invaluable in the success of all 
the components described in this case study. Each course had been structured in a 
similar manner (using the same template) and thus contained learning outcomes, 
defined learning activity opportunities and clear signposts to relevant ROLE wid- 
gets. Thus the use of such structured content made its repurposing as an eBook 
much easier to carry out. Consequently, it was possible to build on the success of 
both ROLE online courses by raising awareness about PLEs and SRL through an 
additional informal based educational channel: the interactive eBook was devel- 
oped. Similarly the eBook reused the previously developed course template by 
providing an introduction to PLEs and SRL as well as giving an opportunity to 
readers to try a selection of ROLE widgets through a set of embedded interactive 
learning activities. 
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Successes and Failures 


The level of success for this case study can be measured in a number of different 
ways. Both qualitative and quantitative data has already been presented that outline 
both the positive and negative impacts that were observed by ROLE presenters or 
recorded by participants in the PUEU survey. Most participants were willing and 
able to take part in the ROLE workshops although some were hesitant to use this 
type of technology. Those people sometimes required encouragement from their 
peers or more experienced colleagues in order that they make progress. Most of the 
free-text responses recorded in the PUEU survey ranged across the possible spec- 
trum of experiences from anxiety to amazement in terms of the potential use of such 
widgets to enable collaborative learning for example. It is fair to say, with this 
supporting evidence, that the case study has been successful in terms of raising 
awareness of PLEs to a cross section of HE staff and groups of research students. 

It would be unfair to state that there were direct failures in this case study. 
Nevertheless, it should be noted that when technology was intermittent in terms of 
access this had a dire influence on the experiences gained by those attempting to 
complete the workshop activities. Frustrations with widgets not fully working or a 
simple breakdown in internet access had a very negative impact on all participants 
but in particular proved to be major stumbling blocks for those who were less 
confident or competent with the PLE or individual widgets. 


Best Practices for PLE Adoption in Informal Learning 


As a result of recording the successes and reflecting on some of the perceived 
failures in this case study, it is possible to list a set of best practices for the adoption 
of PLEs by informal learners: 


¢ Accessible and easy-to-use tools: Best practice in this case study indicates quite 
clearly that a simple format for the ROLE tools is required that enables a range 
of learners to use them effectively and efficiently. 

¢ Multi-format introductory and guidance learning course materials: Learners are 
in need of guided learning materials that will help them understand the value of 
the new technologies. Best practice in this respect included the development of 
the ROLE online courses and the ROLE eBook. 

¢ Tailor tools to meet the needs of specific subject audiences: Best practice here 
should be to enable adjustment or even design for learners studying particular 
subjects or, alternatively, educators researching a wide range of topics to be 
implemented as required. 

¢ Tools that harvest information from appropriate repositories/platforms: A set of 
generic search widgets were fully tested in this case study thus enabling use 
across a variety of learning contexts. As a consequence, some ROLE widgets 
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(e.g. Binocs and ObjectSpot) have been successfully used and repurposed in 
other test beds of the ROLE project. 

e Fostering a culture with a community willing to engage with new innovative 
technology: In order to maximise the adoption of PLEs, a suitable culture 
towards new technologies needs to be fostered. This case study demonstrates 
that the teacher or trainer may also need to adapt their own approach in order that 
they too are receptive to change. 


Conclusions 


The different components of this case study have enabled us to extend our under- 
standing of the potential impact of the ROLE technologies within a wide variety of 
informal learning contexts and scenarios. This case study has numerous rich 
contexts in which there is potential for significant impacts of both PLEs and SRL. 
The challenges faced and lessons learned in each component of this case study have 
also been reported here. As with every new technology, some resistance was 
expected and initially faced in most cases by the participants be they educators or 
managers in HE institutions or likewise e-Learning practitioners elsewhere. Over- 
whelmingly, however, most of the ROLE tools were positively received by those 
who participated in the events, activities, or used the artifacts. Each event, activity 
or artifact that has been explored has also involved a cross section of representative 
individuals. This approach has enabled the ROLE project as a whole to collect 
experiences covering a large variety of learning contexts and requirements. 
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Case Study 4: Technology Enhanced 
Workplace Learning 


Michael Werkle, Manuel Schmidt, Diana Dikke, and Simon Schwantzer 


Abstract Within the ROLE research project it has been evaluated how personal 
learning environments (PLEs) perform in different surrounding conditions. Com- 
panies do often lag behind in the state-of-the-art developments from research 
especially in terms of Technology Enhanced Learning. Festo therefore observed 
on its internal test bed how PLEs can be implemented in business contexts and how 
to involve the learners in this process. Since there is already a broad variety of tools 
to organise and manage formal learning processes within companies the test bed 
didn’t start by scratch either. The focus was thus on how to open up an existing 
learning management system (LMS) towards a PLE. During this process many 
experience from both learners as well as administrators, and training organisers 
have been gathered. One of the lessons learned is that a pure PLE doesn’t fit the 
requirements on personnel development in business context, but certain PLE 
aspects can improve individual learning processes significantly. One showcase is 
the Festo LearningTube which was developed during ROLE. This is an example for 
the successful integration of user generated content into a corporate LMS. 


Keywords Technology enhanced workplace learning * Personal Learning 
Management System * Evaluation « Test bed 


Challenges and Solutions in Technology Enhanced 
Workplace Learning 


The competitiveness of a company depends strongly on the skills and abilities of its 
managers and employees. The development of information and communication 
technologies (ICT) offers a wide range of tools and application options or even 
completely new forms of learning in this regard. Thanks to the development of the 
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Internet in particular, new multi-media learning scenarios and teaching methods 
increasingly complement, compete with and even partially replace classic class- 
room-based teaching. 

While these are usually large companies in the field of formal learning which 
have been able to establish themselves in the area of classic e-learning or blended 
learning concepts on the further education market and in the field of staff develop- 
ment, technologies and learning scenarios for promoting informal learning in the 
workplace are given even less consideration or fail before they have got off the 
ground. 

Why is this so? The idea of setting learning processes in motion or even 
optimising them simply by using new learning technologies for planning, 
performing and reflecting on further education courses, is just as harmful as the 
complete ignorance of such usage scenarios and potential. Not only technical 
requirements, but also media-didactic conception and design aimed at specific 
target groups are crucial for the efficient integration of learning technologies, 
which are usually web-based educational applications. 

Teachers and learners within companies often lack media competence and the 
ability to apply self-regulated learning (SRL). Thus, train-the-trainer concepts for 
these skills are needed. Something that should not be underestimated is that 
innovative approaches often also fail due to the lack of willingness to change or 
poor implementation. 


Trends in Technology Enhanced Workplace Learning 


In the last years, the rapid changes in business environment set new requirements on 
the development of learning methods, as well as learning software and content. The 
experiences of the last years show that the curriculum-based learning approach, 
based on the assumption that the learner “consumes” knowledge in the form of 
predefined learning content, was continuously extended with SRL approaches 
based on the cognitive learning theory. This happened through the active involve- 
ment of the learner into the learning process, that is planning and defining the 
learning strategy, searching for appropriate methods, tools and materials, time 
management, reflection, and undertaking corrective actions. Such characteristics 
as adaptivity, responsiveness, and personalisation became, thus, mandatory for the 
modern learning software solutions. 

Also, the penetration of mobile media, such as smartphones and tablets, into the 
Technology Enhanced Learning (TEL) branch caused significant changes in the 
conception and realisation of workplace learning activities. The possibility to use 
small software applications and pieces of content (“learning nuggets”) became an 
indispensable component of virtual learning environments making the learning 
process time and place independent and optimised for mobile devices. Further, 
the “socialisation” of the web (e.g. the emergence of social networks, social sharing 
platforms, micro blogging, and so on) caused continuously increasing application 
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of social media technology in education, using software tools supporting commu- 
nication, content sharing, joint learning activities, and providing mutual feedback 
between the learners. 

Thus, the following trends/challenges in terms of workplace learning have been 
observed: 


— Increasing networking of the learners (collaborative learning/social learning) 

— Creation of smaller, highly concentrated learning content (micro learning) 

— Increasing importance of mobile learning (“bring your own devices”) 

— Individualisation and self-regulation of the learning processes (PLE approach) 

— Videos as the learning medium with increasing potential 

— Increasing importance of specific (experience-based) knowledge and user-gen- 
erated content 


The ROLE Solution 


While learning, software met the requirement of becoming user-centred, flexible, 
and social, organisational structures needed more time to adapt to the new trends. 
The evaluations conducted within the scope of the ROLE Project showed that the 
vocational training and workplace learning providers appreciated the SRL approach 
and the idea of personalisation of the learning tools and content. On the other hand, 
they wanted to keep control over the learning environment, as well as tools and 
content provided to the end users. Further, it became obvious that the implemen- 
tation of SRL in an organisation needed the development of specific competencies 
by the learners, as well as guidance through the learning process from its very 
beginning. 

Thus, a learning software solution allowing combination of curriculum-based 
and SRL approaches, and providing both standard functionalities and content and 
personalised tools and materials has become necessary. To address this require- 
ment, the ROLE Project developed a Personal Learning Management System 
(PLMS), which is an OpenSocial-based Learning Management System (LMS) 
combining functionalities of a LMS and a Personal Learning Environment (PLE) 
and allowing users to construct their virtual learning environment according to their 
learning history, goals, and preferences. 

The following sections describe the main pedagogical and technical concepts 
underlying the development of the ROLE solution from the point of view of Festo, a 
test bed which actively contributed to the development, testing, evaluation, and 
application of the ROLE approach and technology. The Festo Showcase describes 
(a) the development and application of a learning video sharing portal 
(LearningTube), (b) the extension of the corporate LMS with additional web- 
based learning applications (widgets), c) the evaluation of the PLMS developed 
based on the corporate LMS and including external learning widgets. 
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Technology Enhanced Workplace Learning at Festo 


The Learning Company Festo 


The Festo AG &Co.KG was founded in 1925 by Gottlieb Stoll and Albert Fezer in 
Esslingen, Germany. Initially, it manufactured wood cutting tools and later diver- 
sified into the automation industry. Now Festo is a leading, worldwide supplier of 
automation technology with approximately 30,000 catalogue products, customised 
solutions, ready-to-install automation systems, and a matching range of before- and 
after-sales services. According to the Engineering & Production on-demand con- 
cept, users can adapt these solutions to their individual needs, which actually 
increases the number of products offered exponentially. Furthermore, Festo is the 
performance leader in industrial training and education programs and offers a 
comprehensive range of learning systems for industrial training and education— 
from seminars, training and consultancy in 26 languages to e-learning and complete 
turnkey learning centres. Some 42,000 customers worldwide take part in Festo’s 
seminars or are educated at the company’s own training facilities. In 2012, the Festo 
Group had about 16,200 employees and a consolidated turnover of €2.2 billion. The 
61 independent national companies serve customers in over 250 offices in 176 coun- 
tries worldwide. 

Festo AG & Co. KG defines itself as a “learning company” which would like to 
constantly expand the knowledge and potential of its employees in order to encour- 
age technical innovation and product development. Festo is a globally-oriented and 
independent family company with its headquarters in Esslingen. The largest pro- 
duction and logistics site is in St. Ingbert/Rohrbach, which is home to the cylinder 
production, the Customer Service Center and the Festo Learning Center. 


The Festo Lernzentrum Saar GmbH (Festo Learning Center) 


The Festo Learning Center was founded in 1994 and is located next to Festo’s 
production plant and logistics centre in St. Ingbert/Rohrbach, Germany. As an 
accredited institution for advanced vocational training, the Festo Learning Center 
offers a wide range of personnel and organisational development programmes. The 
service portfolio comprises the whole spectrum of further education (seminars, 
training courses, the private technical school “Festo Technikum”), subsidy consult- 
ing, vocational retraining and qualification, industry consulting, and e-learning. The 
customers are international enterprises of all branches and institutions, as well as 
Festo employees and private persons. 

Within the ROLE project, the Festo Learning Center aimed to define require- 
ments of how to create a ROLE environment within a professional industry 
business. The Festo Learning Center wanted to demonstrate the use and benefit of 
ROLE technologies for the learners by addressing and supporting them in planning, 


Case Study 4: Technology Enhanced Workplace Learning 163 


realisation and evaluation of qualification measures with the help of the web-based 
services on the Festo Virtual Academy—Festo’s global e-learning platform. 


The Virtual Academy 


The Festo Virtual Academy is the central LMS of Festo with a corporate internal 
focus. It is based on the software CLIX developed by IMC (see section “IMC and 
CLIX Learning Suite”). The LMS supports personnel development processes 
within the Festo organisation and offers strategic relevant online courses to the 
learners. 

The Virtual Academy is open for each Festo employee worldwide for their 
personal further education by means of the lifelong learning approach. The LMS 
is accessible via the internet to facilitate SRL processes for the employees (learning 
independent of time and place). In addition to the self-learning offers, several 
blended-learning modules are available. The web-based trainings (WBTs) are 
designed according to didactic models in order to make their “consumption” as 
easy as possible for the learners. 

The Virtual Academy has more than 9,000 users distributed all over the world, 
more than 80,000 logins p.a., more than 800 learning contents in different strategic 
learning categories with a total learning time of approximately 2,000 h. The 
contents are provided mostly in German, English and Spanish. For specific topics 
additional languages like Chinese, Japanese, Portuguese and Russian are also 
available. 

In the ROLE project, the Virtual Academy was one of the five original test beds 
and addressed the issue of providing a responsive learning environment within 
further education activities in a company. This included not only continuous 
technical and media-didactical possibilities of a LMS and the content within, but 
also the technical possibilities beyond the LMS approach. 

The application of ROLE at Festo led to better support of individual learning 
processes (in terms of self-regulation) and also better support of collaborative and 
social learning in a company. Therefore, informal learning increases its importance 
for the workplace learning and also the possibilities to record, share and save this 
expert knowledge within the company. 

The following chapters describe the most important TEL-topics for the current 
and future developments of the Virtual Academy as well as the ROLE project with 
the Festo test bed, especially 


— The initial situation 

— The project vision 

— The specific target group 

— The characteristics of the business context 

— The main organisation related challenges and the requirements 
— The technical implementation 

— The evaluation of the project results 
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Trends of Technology Enhanced Workplace Learning at Festo 
Making Learning Environments More Personal 


The PLE approach is nothing particularly new, but is still a vital research topic in 
the TEL-community. PLE stands for a PLE, i.e. an individual composition of 
learning services that helps learners to plan and conduct their learning activities, 
and to reflect on the learning process and progress. Considering the fact that 
learners are often a very heterogeneous group (with different learning needs and 
goals, learning styles, individual learning experiences, knowledge, learning prefer- 
ences or different job roles), the potential of the PLE approach is undisputed. 

In 2007, Graham Attwell (2007) defined a PLE as being “comprised of all the 
different tools we use in our everyday life for learning”. Considering this quote 
alone, one can be of the opinion, that in business contexts, especially in e-learning, 
every employee has already developed a PLE consisting of standard and company- 
specific software as a digital toolbox. But far from it, at the moment the PLE 
approach is more theory than it is implemented in further education scenarios or 
personnel development processes in companies. It seems that further education is 
more and more lagging behind technical developments. 

The main hurdles for the TEL implementation in companies are: 


— At the moment, learning processes in companies are unilaterally strategic driven 
in a top down manner. 

— Most of the learning processes are formal. The potential of informal learning and 
user generated content is not being exploited. The paradigm change of content 
towards the expert as author is not yet turned into practice in companies. 

— Companies have already made big investments in learning technologies (perhaps 
in e-learning content or an LMS) and promote only these. 

— Frequently there are a lot of existing internal learning tools in (bigger) compa- 
nies, but these tools are often not synchronised. Therefore, the promotion of new 
learning tools across departments is very difficult. 


Consequently, the Festo test bed examined in ROLE how the PLE approach can 
enhance learning opportunities for employees. The target was not to create a “pure” 
PLE, but rather to combine advantages of a PLE with an existing LMS. Thus, it was 
not a question of replacement or substitution of an existing, traditional LMS in a 
company but rather an approach that enabled the enhancement of their current 
delivery mechanisms. It was also a question of how an existing LMS can be 
enriched with new information and communications technologies that accordingly 
enhance the end-user (the learner’s) experience. This will be explained in detail in 
the section “Implementation of the Personal Learning Management System”. 
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Videos as the Learning Medium of the Future 


The power of the image is used on the Internet to disseminate information quickly 
and easily. Photos and graphics are omnipresent in the worldwide web today—be it 
for advertising or promotional purposes, to inform or simply to spread a bit of 
happiness. Today’s technology makes it very easy for each and everyone of us to 
create pictures or videos and publish them on the Internet. Individual users already 
actively use video production and sharing to exchange experiences and knowledge. 
The trend towards video-based learning is being adopted by private companies and 
higher education. 

Pioneers in this area include the Khan Academy’ with a thematic collection of 
over 4,100 instructional videos and Stanford University, which operates its own 
YouTube channel.” However, the YouTube channel alone is not the impressive 
fact, but those 215,289 subscribers downloading over 55 million videos on demand, 
illustrates its reach. The private online academy Udacity,’ founded in 2012, pro- 
vides similar figures. The philosophy here is “Learn. Think. Do.—Higher Educa- 
tion for Free”. The first two free video-based online courses alone reached 90,000 
participants (Lewin 2012). 

The key to the success of these video tutorials is the didactic simplification 
through visual representation. Wherever one previously had to describe a subject in 
words, written or verbally, or how complex product drawings had to be created, 
today it is simple to visualise information using moving, narrated images. This 
usually saves the knowledge carriers and producers of video tutorials more than just 
time in comparison to creating professional teaching materials or specialist articles. 
The creator of a video tutorial is able to use the sound and video track to give his 
“knowledge” not only a personal note but also a kind of personal signature. 

The knowledge content is closely connected to the knowledge carrier and these 
are not as easy to separate as in the case of pure text-based content. Thus, 
statements, opinions and experiences can be expressed personally through inter- 
views and the interviewed person “signs” it with his name. Ultimately, the web 
video medium suites often better to meet the need for communication and self- 
expression than “pure” text. From the learner’s point of view, particular application 
or process knowledge is taken in more easily via pictures and transferred to “real 
life” than when if it is read from texts. The recipients also use their auditory and 
visual sensory paths to process information, which helps to anchor it in the memory. 

Festo has recognised this trend towards instructional videos. It developed an 
award-winning tool, the LearningTube and Recorder, as part of the ROLE research 
project. The LearningTube provides all Festo employees with a simple way of 
creating video tutorials and sharing them with colleagues in the company in order to 
document best practices arising in everyday work and make them available for 


1 http://www.khanacademy.org/about 
7 http://www.youtube.com/user/StanfordUniversity 
3 https://www.udacity.com/us 
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colleagues, successors, and new employees. The innovative aspect of that is the 
combination of the LMS with easy-to-use authoring software which allows content 
owners as well as every learner to instantly create learning content on their own. 
Compared to traditional LMS that only had “time-consuming” text-based authoring 
tools included, the screen recording capabilities of the LearningTube make it much 
easier for the authors to capture their knowledge and transform it into learning 
content. The reason for the perceived simplification is that experts can easily reuse 
the tools and media they use in their everyday life to transport the information such 
as PowerPoint slides or recording the software they need to explain. Thus they do 
not need to develop new skills, the system is built on the skill-set the experts 
already have. 


The ROLE Project and the Festo Test Bed 


Project Vision 
Learning in the Virtual Academy Before ROLE 


The learning processes at Festo can be described as self-controlled by the learners 
(employees). The users browse offered learning programmes in the catalogue of the 
Virtual Academy and select those they need. During the learning process itself, the 
learners can decide by themselves what they would like to learn and which parts of 
the web-based training content they would like to skip. Since the platform is 
available online, the learners can even access the Virtual Academy from home 
and participate in trainings on a voluntary basis. The learners are free to define and 
plan what and when to learn, and how frequently to work on the web-based training 
offered in the catalogue. 

The vision of the ROLE project was to improve especially the aspects “respon- 
sive” and “open” of the already existing learning environment of Festo—the Virtual 
Academy as the central LMS of the company. From Festo’s point of view the main 
targets were to improve existing learning systems according to: 


— Openness and adaptivity 

— Communication with other learners 

— Facilitation of collaboration and peer-assisted learning 

— Switch collaborated and individual work 

— Exploring ways of benefiting from the experience acquired in a company 
— Best practice sharing 
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Challenges and Requirement Analysis of the Test Bed 


The Festo Learning Center aims to meet the requirements of learning environments 
within a business context. To achieve this aim, Festo effectively opened up its 
platforms and tools to “mix and match” and be interoperable through the use of 
ROLE tools and technologies. 


Organisation Related Challenges and Requirements 


Since the Festo Virtual Academy is effectively a corporate learning environment, 

some special surrounding conditions have to be considered within the ROLE 

process. These special “challenges” depend on the fact that a corporate LMS has 

one central main function: the further development of personnel in the company. 
Issues relating to these surrounding conditions are, for example: 


— The uncertain scope of openness of a corporate learning environment. How wide 
can a corporate learning environment be opened up? 

— The “job role” of the learner has to be in the focus of all learning processes 

— Knowledge sharing is harder to realise in the job context than in non-working 
life 

— Understanding learning processes during daily work and after work 

— System restrictions and data security 


Target Group Related Challenges and Requirements 


The Festo test bed is focused on LMS users and especially LMS users in a company. 
This target group has special needs and the surrounding conditions in companies are 
not as flexible as those predominant, for example, in universities. In contrast to 
students, business learners are a very heterogeneous target group with big age 
differences (from 16 to 65), different educational backgrounds and previous knowl- 
edge, job-roles, learning requirements, learning preferences and learning goals. 

Further, the learners in business environments have primarily to fulfil their job 
role and learning is mostly to support them in doing so. Due to high workloads, it is 
often hard to learn on the job or in other words, there is no or just little time for 
learning available. It is often not so easy to disengage workers from their daily 
practices. Thus, learners at the workplace need to be supported systematically, not 
only with new TEL solutions, but also with their development goals, the working 
and learning conditions in general and their work life balance. 
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User Related Challenges and Requirements 


There is one more aspect to be considered, which is related to the personal 
requirements of the learners. The learners need a set of specific skills, the so-called 
“self-regulated learning” (SRL) skill-set. These are skills the learners must have to 
be able to successfully plan, conduct, and evaluate their learning activities. Some of 
these skills might be new to some learners, whereas others might be present and 
used already, but the awareness about that fact is still missing. 

To address this requirement, Festo initiated the “Fit for Self-Regulated Learn- 
ing” initiative in the ROLE project. The plan was to implement various SRL 
learning modules in the offered ROLE service. Some of the modules should 
explicitly show that they support learners in getting SRL known as a method. 
Others would be implicitly woven into ROLE services to make their impact on 
SRL visible, for example within wizards. 

Putting the idea behind “Fit for self-regulated learning” in a nutshell, one can 
state that SRL should improve the learning outcomes. This is realised by offering 
the learners SRL as a manual. This would not be a technical description of the 
platform, but rather a manual for learners that shows how to learn effectively and 
successfully in the ROLE environment with the help of SRL. 

Festo considered the benefit of the approach to be very high and therefore 
decided to personalise learning through ROLE and made learning more demand- 
and service-oriented for the users of the Festo Virtual Academy. 


Implementation of the Personal Learning Management 
System 


IMC and CLIX Learning Suite 


IMC is currently Europe’s largest learning management solution provider that was 
established in 1997 by Prof. Dr. h.c. August-Wilhelm Scheer as a spin-off of 
Saarland University. IMC offers a comprehensive portfolio of e-learning-related 
products, supporting all business processes in training and education. IMC also 
provides professional services covering the whole value chain of content design and 
production as well as consulting and managed learning services helping clients to 
(re)organise their learning processes and to select, implement, adapt and integrate 
suitable software systems and technologies. 

IMC’s product portfolio includes professional authoring and content recording 
tools (Content Studio, LECTURNITY), a Business Process Guidance System 
LIVECONTEXT, as well as a Learning and Talent Management System CLIX 
supporting HR- and Personnel Development processes at organisations. CLIX 
contains two main modules: the Learning Suite, which is a LMS in the classic 
sense, and the Talent Suite supporting Talent Management processes, such as 
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Talent Identification, Competency Management, Career and Succession Planning, 
Performance Management, etc. 

In scope of the ROLE Project, IMC used CLIX Learning Suite to implement the 
concept of the Personal LMS. CLIX Learning Suite in its basic configuration 
includes such functionalities as Organisation Management, Process Management, 
Content Management, Testing and Assessment, Resource and Capacity Manage- 
ment, Report and Compliance, Evaluation, and some others. The Social Learning 
module contains communication tools, such as chats, communities, wikis, and 
forums allowing making use of the synergy effects of collaborative learning. 

In order to increase personalisation of the system and to extend collaborative 
functionalities, CLIX Learning Suite has been integrated in scope of the ROLE 
Project with an OpenSocial PLE, thus, constituting a so called Personal LMS. The 
PLMS aggregates learning resources and applications available in the web and 
selected by the users. Its structure aligns phases of the SRL process and assists the 
users in planning of their learning activities, search for learning content and tools, 
training and testing, as well as in reflection and evaluation of the learning progress. 

Addressing the increasing need in acceleration of the knowledge acquisition due 
to rapidly changing organisational and technical conditions at organisations, and in 
personalisation of the learning process increasing employees’ motivation and 
effectiveness of the training measures, an adaptable learning environment 
supporting workplace learning had to be developed. The identified requirements 
from the user side were: firstly, for the learner’s point of view, to make the system 
customisable in terms of tools and content according to the current learning needs, 
and secondly, from organisational point of view, to keep control over the learning 
environment to ensure transparency of the learning process and the achievement of 
planned learning results. 


From LMS to Personal Learning Management System 


The early development of LMS is aimed at coordinating the learning processes in 
organisations in terms of training measures participant and content management, 
including creation of course curricula, adding learning materials and tests, as well 
as providing meta-data such as course dates and place, procedure of entering the 
course, course completion and certification. Over the last years the LMS developed 
in the direction of the Human Resource Management Systems (HRMS) providing 
functionalities for personnel assessment, competency management, learning and 
development, as well as succession planning. While increasing the planning and 
organisation capacity of the LMS, the learning process supporting infrastructure has 
not changed much. 

The new learning approaches, such as SRL, collaborative and social learning 
have been realised in PLEs allowing the users to construct their learning environ- 
ments themselves by selecting learning tools and content in a way supporting 
acquisition of the desired knowledge. In scope of the ROLE project, a PLE is a 
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web-based infrastructure, where the users can access, aggregate and manipulate 
learning applications and resources of their preference, as well as communicate 
with other users sharing experiences and collaborating on projects (Overton 2009). 
Importantly, the PLE uses web widgets, which are small web-based software 
applications, to support particular learning and teaching goals or training of some 
specific skills. 

In order to create a software solution supporting both approaches, an OpenSocial 
PLE has been integrated into LMS combining functionalities of both systems. Thus, 
the PLMS provides instructions and pre-defined learning materials allowing the 
users to complete learning courses as usual, but also it ensures the learning process 
autonomy offering personalised learning spaces, in which the users can add and use 
additional applications and resources. Further, the PLMS allows browsing addi- 
tional learning content directly out of the system with help of the Media Search 
Widget, thus, being more open than classic LMS. Also, it offers guidance through 
the SRL process and support to its each phase, which are described in the sections 
below. 


Psycho-Pedagogical Integration Model and Personalised 
Learning Spaces 


The OpenSocial directory of the PLMS is structured according to the phases of the 
SRL process described in the Psycho-Pedagogical Integration Model (PPIM) 
developed in the ROLE Project (Fruhmann et al. 2010). This model identifies 
four main steps: 


1. Plan: This phase includes definition of the learning strategy, learning goals, 
actions to be taken to achieve these goals, as well as preferences in the sense of 
tools and types of content that will be used. 

2. Search: At this step the learner searches for learning resources and tools within 
the learning environment and outside of it. Here, the user may get recommen- 
dations from tutors and peers, but also use recommendation systems to find 
appropriate resources. 

3. Learn: Learning includes studying of the selected tools and materials, attaining 
skills, training and testing, as well as assessment by tutors and self-assessment. 

4. Reflect: This phase implies gathering feedback from different sources and self- 
evaluation, as well as reflection on the learning process and achievements in 
order to evaluate the usefulness of the learning strategy and particular actions 
and their correction if needed. 


These phases are summarised as “plan-search-learn-reflect” loop (see Fig. 1 
(Modritscher and Nussbaumer 2012). 

The OpenSocial directory of the PLMS is divided into four learning spaces 
corresponding to these four steps. In order to provide necessary guidance to the 
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Fig. 1 Psycho-pedagogical integration model 


users, each learning space is populated with pre-defined learning widgets. These are 
Open Source tools that were developed by the ROLE project or found in the web 
(all rights reserved) and reviewed concerning their suitability to support one or 
more of the learning phases. Further, the PLMS contains a list of additional tools. 
Thus, the users may use pre-defined applications and/or supportive tools from the 
list arranging them in the learning spaces. This allows an efficient integration of 
external tools into the system respecting the interests of both the learners and the 
organisation (Fig. 2). 

Besides learning applications, each learning space contains an introductory 
video to the respective phase of the SRL learning process and a Wizard Widget 
containing supporting questions and hints to assist the user in getting started with 
using the system. The user acceptance evaluation showed that the availability of 
these assisting tools was highly appreciated by the users (see Evaluation section). 
Thus, the PLMS aims at providing as much guidance as necessary, and as much 
assistance as possible (Schanda et al. 2012). The development of the PLMS and its 
technical implementation focused on the personalisation, adaptivity, and user- 
friendliness of the PLMS making it responsive to the needs and preferences of 
the users. 


Technical Implementation: OpenSocial LMS 


In scope of the technical implementation, two major aspects were identified in order 
to extend the CLIX Learning Suite to be used as PLMS: on the one hand, a 
mechanism was required to enable the integration of external components into 
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Fig. 2, ROLE PLMS search learning space 


the CLIX portal. These components had to be implemented using a standardised 
technology for the following reasons: 


— In order to separate the development processes and technologies for components 
of PLMS from those of CLIX. 

— To enable external developers to create components without technical knowl- 
edge of the environment they are embedded in. 

— Inorder to allow integration and re-use of existing components, e.g. components 
which are provided in the repository. 


On the other hand, an interface was required to exchange social information 
between PLMS components, CLIX as a hosting environment, and external services. 

Taking those requirements into consideration, the decision was made to use 
OpenSocial* as an open cross-platform standard specifying a REST-API to 
exchange social information. Having the API specified, the technology stack was 
completed by supporting the Google Gadget specification? to create components, 
further on called widgets. Widgets are applications designed to only take small 
parts of a screen and make use of web technologies (HTML, CSS, and JavaScript). 
For the communication between the widget and the environment, a JavaScript API 
was provided, which also included the interfaces of the OpenSocial API. 


4 http://opensocial.org 
5 https://developers.google.com/gadgets/docs/spec 
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The implementation of the OpenSocial API and Google Gadget specification 
was based on the Apache Shindig project.° Apache Shindig was realised as a 
standalone web application, which was connected over a dedicated web service 
interface with the CLIX Learning Suite. This architecture allowed separated main- 
tenance processes for both platforms. Shindig took two roles within the PLMS: as 
rendering server for widgets and as OpenSocial API provider. CLIX acted as data 
provider and environment for embedding the widgets rendered by Shindig. 

At the next step, an integration point for the widgets inside the CLIX portal had 
to be found. As the PLE structure does not necessarily depend on a concrete course, 
the integration was realised using dashboard pages. Dashboard pages were designed 
to create a mash-up of panels showing user-centred information (e.g. a course list, 
news, social media updates, etc.) and therefore built a capable environment for 
widgets. The implementation aimed to provide a seamless integration with existing 
dashboard features. A new panel type was introduced and each instance of this 
panel represented a single widget. 

Platform administrators can select which widget panels are available on a 
dashboard page and can pre-configure the widgets. They can choose if a widget 
has a system-wide or a user-specific configuration. In the latter case, the users have 
their own configuration interface for the widget directly integrated in the 
dashboard page. 

The final layout and content of a dashboard page allow nearly any degree of 
personal individualisation. Whilst the pool of available widgets is previously 
defined, their visibility, application, and position can be made customisable for 
the individual user. The range of self-controlling is therefore very large; both 
settings of fully pre-configured widget mash-ups and open-space solutions, where 
users combine widgets starting from a blank page, are possible. Learning processes 
can also be supported by combining multiple dashboard pages, covering either 
different steps in a learning process or different topics addressed by specific pages. 


Concept and Evaluation of the PLMS at Festo 


The Virtual Academy Case study 


This case study reports on how the ROLE environment can be open to a mix of 
internal learning applications alongside external ones. This is regarded as a key 
success factor for project ideas that emerge from developments like ROLE to 
influence the promotion of further education in companies and meets the overarch- 
ing premise of this case study namely that it demonstrates “an internal learning 
opportunity in a company” environment. In essence, this section, thus, presents an 


6 http://shindig.apache.org/ 
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evaluation of a case study relating to the Festo Virtual Academy and demonstrates 
how internal learning opportunities can be improved in a company. 

This case study therefore sets out to address the issue of providing a more 
responsive learning environment within the further education activities of Festo. 
As previously indicated, to achieve this aim, Festo opened up its e-learning 
platform to be interoperable and open for mixing and matching with ROLE tools. 
This combined approach was called a “Personal Learning Management System” 
(PLMS), where the PLMS was comprised of the LMS and PLE together, hence 
when combined resulting in the acronym PLMS as technically described in section 
“Implementation of the Personal Learning Management System”. 

The actual implementation of this approach, i.e. the degree to which these PLE 
enrichments are embedded, ultimately depended on both company-specific require- 
ments and on the individual learning preferences, as well as the anticipated learning 
experiences of individual learners. In general terms, however, the main targets of 
the PLMS approach were: 


— Simplified access and advanced search of relevant content and learning 
materials. 

— Support and improvement of the planning of learning, incorporating the reflec- 
tion phases of the learning activities. 

— Enabling learning motivation and promotion of SRL as well as different forms of 
cooperative learning. 


To achieve this improvement, Festo supported ideas and the development of 
prototypical widgets and tools, which can enhance the learning processes in such 
corporate learning environments. Initially, the first item to be addressed during the 
implementation of the ROLE approach was to improve the “Openness” of the 
system. 

At the first step, a federated search widget was developed together with two 
other ROLE partners, namely IMC and KU Leuven. The idea of the widget was to 
enable a more focused search facility in the Festo LMS. It was developed to search 
learning content in several external online resources and feed results back directly 
to the learner within the Festo LMS. A media-list widget, that was interoperable 
with the federated search widget, was developed too. This allowed users to create 
media lists out of the resources found with the search widget. Both widgets are 
featured in the Festo Virtual Academy. 

At the second step, another perspective of openness considered in the context of 
this case study was the integration of user generated content as well as encouraging 
the possibility for learners to produce content on their own. To achieve these 
outcomes, a commercial screen recorder facility was embedded as a widget, 
which enabled learners to create their own videos. This widget bundle was given 
the names of “LearningTube” and “Recorder”, and together both widgets provided 
all learners, trainers and experts with appropriate support for exchanging and 
communicating training content on a daily basis. 

In this case the Recorder allowed users to create a video, which could then be 
uploaded to the LearningTube and shared with Festo colleagues worldwide. Both 
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Fig. 3 The Festo LearningTube 


tools supported bite-sized learning and ensured that content could be published and 
distributed quickly. This style of rapid e-learning, therefore, allowed users to 
enhance their presentations with the addition of a voiceover and optional webcam 
video. The Recorder widget also provided help and support for creating resulting 
screenshots, enabling users to add their own commentary to PowerPoint presenta- 
tions or even enabling them to emulate software simulations. 

The LearningTube widget bundle was acknowledged by the Comenius- 
EduMedia organisation, which gave an award for the practical application of 
educational, thematic and design excellence in educational media. Festo was the 
only industrial enterprise to receive such an award. Equally, the widget has offered 
to Festo employees around the globe, an interactive educational video channel that 
has also become a valuable learning tool. As a result of this implementation, there 
have been over 220 video uploads by Festo employees since July 2011, consisting 
mostly of screencasts and recorded presentations. These videos have been accessed 
over 15,000 times to date (April 2013) (Fig. 3). 


Evaluation 


The concept used to evaluate the ROLE solution at Festo consisted of a combina- 
tion of questionnaires, selected expert interviews, a focus group for requirement 
gathering purposes, a taskforce observation and interviews by project members. 
These various elements ensured that there were standardised frameworks for 
evaluation and also personal contacts to offer possibilities to clarify confusing 
questions and allow a little more depth. 


176 M. Werkle et al. 


During the project, Festo carried out two main evaluation loops. To this end, 
Festo founded a small focus group of 26 colleagues to test and evaluate the ROLE 
technologies in the Virtual Academy from the learner’s perspective. Qualitative 
data about the usefulness, usability, look and feel, the innovativeness, the relevance, 
the clarity and the improvement potential of the presented ROLE technologies were 
collected. Therefore, the first evaluation loop was conducted by a specific ques- 
tionnaire, whereas the main focus of the second evaluation loop was to observe how 
the test persons work with the learning technology. 

The use of the focus group meant that although the evaluation was not on a large 
scale, the results were of high quality. This was due to the expert knowledge, the 
aforementioned method mix and the possibility to read between the lines and to 
receive more detailed feedback. 


The First Evaluation Loop 


As described, the first step was to implement two widgets in the existing Festo 
Virtual Academy LMS, thus, enriching it with appropriate PLE elements. The first 
evaluation loop consisted of a questionnaire about the developed media search and 
the media list widgets. The evaluation of these widgets was not possible with the 
existing standard questionnaire of the virtual academy. Therefore, a new question- 
naire, asking about personal information, preferred forms of further education, daily 
use of the internet, affinity to Web 2.0, benefit of web 2.0 and the described widgets, 
was created. This questionnaire was emailed to the members of the previously 
described focus group; a screencast introducing the ROLE project, the ROLE 
approach, and the developed widgets was attached. 


Results 


The response rate of 61.5 % (16 of 26) was not as high as expected, but it should be 
kept in mind that the members of the focus group had to give priority to their normal 
jobs and the effort required in the different evaluation loops had to be reasonable. 
The focus group supported the Festo project team on a voluntary basis partly in their 
leisure time. So their resources and feedback were important, but also very limited. 
All responses to the questionnaire in this case study regarding the look & feel, 
usability and perceived usefulness were very positive concerning the applied ROLE 
approach in the business context. It showed that 63 % of the users liked the look & 
feel as well the usability. 

Nineteen percent said that that the look & feel should be improved and 13 % 
stated that the usability should be improved. Regarding the quality of the search 
results, the performance and the fun factor of the widgets, 38 % of the users said 
these issues were in need of improvement. Forty-four percent rated the quality of 
the search results as good, the performance got a good grade from 50 % of the users, 
the fun factor was rated as good by 38 % and even as outstanding by 13 % (Fig. 4). 
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Fig. 5 Perceived usefulness of the prototypes 


Evaluation of the benefits of the prototypes showed that most of the users saw a 
high or very high benefit of the offered tools. The highlights regarding the benefits 
were in this case the federated search feature over several knowledge resources. 
Fifty percent stated a benefit and as many as 38 % stated a high benefit. For the 
rating feature 31 % stated a benefit and as many as 44 % stated a high benefit. The 
good overall impression of the evaluation reflects especially in the recommendation 
value—in total 88 % of the test users would recommend the tools to their col- 
leagues. Last but not least, perceived usefulness and effectiveness was evaluated 
with the question “do you think the offered services will help you to work more 
effectively in your job then at present?” Thirty-one percent fully agreed, 50 % 
agreed and only 6 % denied that they would be more effective when working with 
these tools. This was a really good result for this early prototype evaluation from 
Festo’s point of view (Fig. 5). 
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The Second Evaluation Loop 


The subject of the second evaluation loop was the PLMS prototype, which was 
developed with the goal of integrating the approaches of the LMSs and the PLEs, 
and thus of supporting the user during SRL. The evaluation was conducted on a 
PLMS prototype, which was integrated into an IMC test environment. 

The acceptance of the learning environment was and is an essential aspect in this 
regard. In particular, if an introduction of training offers, which will be used on a 
voluntary basis, is concerned, it is important for the training provider to determine 
the acceptability extend of the learners. Thus, the objective of the evaluation was to 
find out how well the PLMS was accepted by the employees, and which measures 
could be implemented to further increase its acceptance. 

The following usability factors were important: 


— The selection and addition of widgets, and navigation learning spaces 
— The use of resources, such as wizard widgets and tutorials 

— Acceptance of the PLMS 

— Handling of the PLMS 

— Comparisons with conventional training media and forms of learning 


Therefore, several methods were used in order to evaluate the PLMS. Observa- 
tion was conducted on the one hand, and the so-called “think aloud” or “question 
asking” method was used on the other. In addition, the questionnaire entitled 
“Perceived Usefulness and Ease of Use” was used in this evaluation supplementing 
data obtained by means of observation and the “think aloud method” (“TAM”). For 
reasons of comparability, the questionnaire was developed within the framework of 
the ROLE project, which was intended to ascertain usefulness and user-friendliness 
of the project results as perceived by the learner. It was also used in other test beds. 

The “TAM” was used as a supplementary method in addition to observation. In 
particular, this method was used for examinations of the user interface because it is 
especially well suited for detecting problems of this sort. With the use of “TAM”, 
the thought processes of the learners were investigated while they were dealing with 
the PLMS. 

Test persons were prompted to describe their actions and thoughts out-loud 
during interaction with the learning environment. In this way, the test persons 
were not only able to address any problems they were experiencing with the user 
interface, they could also explain them and, as a result, could reveal design defects. 
Furthermore, the test persons expressed thoughts of satisfaction, enthusiasm or 
motivation when commenting on their actions. Thus, data compiled by means of 
“TAM” provided in-depth knowledge, not only regarding the actions of the test 
persons, but also about their attitudes as well. If applicable, information about their 
experiences gained in dealing with the software, could also be gathered. 

The “TAM” was used in a slightly modified form, which was more comparable 
to an interview technique, by means of which the study director asked the user 
targeted questions regarding comprehension, the sequence of the learning activities, 
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etc. This counteracted the danger that the test person talked less and less while 
working with the learning environment, and no longer remembered to comment on 
his actions out loud. This problem might have occurred because working with the 
PLMS represented a very new and complex task, for which the user required a large 
portion of his cognitive capacity. 

The evaluation supervisor spoke with the test persons or helped them work 
through the learning steps in the PLMS, if requested to do so. However, this 
influenced the experiences, the attitudes and the actions of the learners. This 
influencing factor was taken into consideration in the evaluation of the results. As 
a rule, “TAM” was conducted with help of audio or video recordings. Instead of a 
reconstructive form of data collection, “TAM” made use of so-called registrative 
data conservation. This simplified evaluation and assured the reliability and validity 
of the compiled data. “TAM”, thus, effectively compensated the weaknesses of 
participative observation. 


Evaluation Procedure 


During the evaluation phase 11 interviews were conducted with employees of the 
Festo Learning Centre. The employees came from different departments and had 
different educational backgrounds. The evaluation was allotted a duration of 45- 
60 min per test person. The users were requested to complete a task in the PLMS to 
this end. Subsequently, the test persons evaluated the PLMS with regard to accep- 
tance, system performance, required effort and user-friendliness, as well as use of 
and satisfaction with the application assistance. 

The survey “Perceived Usefulness and Ease of Use” consisted of two parts. 
Those questions which were considered as learning premise, namely those regard- 
ing age, sex and TEL experiences were presented before the PLMS test. The other 
part of the survey dealing with the evaluation of the product, in this case the PLMS, 
was conducted after the PLMS test. 

The evaluation supervisor was available while the questionnaire was being filled 
out, and was able to help the test persons with any questions or uncertainties. 
However, the test persons were initially only asked to respond to the closed 
questions. Then the observation started, during which time the observer took 
notes. If it was needed and time allowed it, the observer went deeper with the 
open question section in order to clear up any unanswered points. 


Results 


The evaluation of the questionnaire entitled “Perceived Usefulness and Ease of 
Use”, provided the following summary: Eighty-one percent of test persons rated the 
learning environment PLMS as useful. Fifty-five percent indicated that they would 
achieve their learning goals somewhat more effectively with such a learning 
environment. The usefulness of the learning environment was rated diversity. 
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Fig. 6 Results of the questionnaire “Perceived Usefulness and Ease of Use” 


Eighteen percent chose “fully correct” and 36 % rather correct while 27 % were 
undecided and 18 % said that the uses were rather not easy. 

There was a strong agreement with 100 % that the use of the learning environ- 
ment was not frustrating. But the evaluation showed that there was still a need for 
improvement. Almost half of the interviewees (45 %) said the use of the learning 
environment was strenuous or rather strenuous. However, the vast majority with 
72 % of the respondents would use or rather use the learning environment, while 
only 9 % said that they rather not use the tested learning technology. The results in 
details were shown in the Fig. 6. 

Further, through the use of the “TAM” it was possible to document positive as 
well as negative statements of the probands. Figure 7 represents a compendium of 
the most important statements, divided into the categories effort, performance and 
facilitation. 

In conclusion, the evaluation results of the PLMS allowed the following 
statements: 


— The use of the PLMS is deemed highly beneficial 

— The PLMS supports the achievement of individual learning goals 

— The PLMS would be used by the learners, if access to a tool was available 

— Usability, as well as the look and feel of the PLMS prototypes, must be greatly 
improved 


In terms of challenges, however, some technical hurdles appeared regarding the 
usability as well as look and feel of the PLMS for future implementations. These 
challenges related to specific computer-based issues and are all undergoing further 
investigation. 
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Fig. 7 Statements from the thinking aloud interviews 


Nonetheless the feedback from the evaluation investigation remains very posi- 
tive. It appears that people really liked the PLMS approach. Additionally, and since 
the test phases took place, albeit on a prototype, the interviewees asked also for 
refinements of the system in relation to their user experience. 


Conclusion and Outlook 


The Future of TEL in Business Context 


The ROLE approach and the gained experience showed the potential of TEL 
solutions not only in the higher education field, but also in business context. This 
sounds quite simple at first, but it is extremely important. Higher education settings 
and learning at a company are not two different worlds, but completely different 
learning scenarios, with different learning goals and needs, different learners, 
different learning conditions and learning possibilities. Often in research projects 
initial solutions are presented without entirely knowing the specific surrounding 
conditions. 

But these kinds of solutions are more innovative than useful. There are a lot of 
hurdles and specifications in companies according to organisational, technical and 
personnel requirements. Therefore, it is important for TEL researchers to start every 
TEL activity with a target group-specific requirement analysis and after this to 
transfer the acquired scientific developments into these new learning environments. 

On the other hand, for bigger companies it is important to be open to new ideas, 
opportunities and learning approaches as that can help to identify, share and archive 
important knowledge in a better or additional way than before. Often a big revo- 
lution is not necessary. Companies have a lot of different learning technologies and 
learning possibilities that are, however, separate, internal closed systems without 
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synchronisation and different responsible departments and responsible persons. 
Thus, these learning technologies are in internal and in external competition with 
one another and a systematic integration of technology-enhanced learning tools is a 
challenge. 

During the ROLE project, Festo has used its chance to learn from the ROLE- 
consortium and bring new research results together with specific business require- 
ments. During the project the Virtual Academy was opened up through enhancing 
CLIX by OpenSocial widgets. The different evaluation loops showed that is not 
only a current trend move away from closed LMS to more self-regulated and 
individualised learning. Rather, it confirmed the first impression of the project 
team to combine the advantages of the existing LMS with the PLE approach. One 
outcome of this approach was the Festo LearningTube which enables every Festo 
employee to produce and share user (expert) generated learning videos on the LMS. 

The good thing is not only the received Comenius award but also the fact that it 
is possible to share very easily user generated content in a company now. The 
knowledge carriers produce their content on a voluntary basis without the necessity 
of additional incentives. This supported Festo significantly in improving this 
learning environment with the developed technologies and the intelligent combi- 
nation of formal and informal learning. 


What Are the Next Steps? 


The Virtual Academy, especially the LearningTube will be continuously improved. 
In this context three topics will be more and more important: 


— Learning analytics 
— e-Learning goes mobile 
— Full text search in videos 


Learning analytics is a very interesting field for Festo in terms of learning 
transfer analysis. Researching how this process can be taken beyond the step of 
e-tests and e-evaluation is a specific interest of Festo. Just in time analysis of user 
data, visualising learning processes to users, context aware services and individual 
recommendations based on learning goals or learning needs would be one of the 
future scenarios. 

When Festo started with the Virtual Academy it was only accessible via the 
Intranet, so people could only learn from their workplaces. This was considered as a 
bottleneck for the learning processes, so the platform was moved on a server that is 
also accessible via the Internet. The goal is to make use of the benefits of e-learning 
especially learning independent from time and location. This step, taken in 2005, 
can be taken even beyond by bringing content also to mobile devices such as mobile 
phones and tablets. This opportunity should be realised as soon as possible, 
especially because the demand of the learners for mobile learning services is 
continuously growing. A big lever for this approach is that the videos from the 
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LearningTube are mobile compliant and can therefore be easily “consumed” from 
mobile devices. 

Video-based learning is more and more an essential source of knowledge and 
information in companies. As described before, the Festo LearningTube is a rapidly 
growing internal video portal. At present there are more than 220 expert generated 
videos and more are being produced every day. Unfortunately the search function is 
not yet as developed. The growing number of videos results in a growing need for 
state of the art search functionalities to support learners as much as possible find the 
learning content they need. The objective is to find a solution that efficiently 
combines voice recognition with the videos on the Virtual Academy and make 
them thereby “full text searchable”. 


Conclusion 


The ROLE philosophy of Festo was to open the existing Virtual Academy LMS and 
extend it with new technologies supporting individualisation of the learning process 
and increasing its efficiency. The LMS and PLE approaches were combined 
towards a PLMS fulfilling existing business requirements. This was technically 
realised by means of integration of the PLE elements into the LMS platform and 
pedagogically with the use of SRL method. As previously described, the first step 
was to implement two widgets (media search and media list widgets) in the existing 
Festo Virtual Academy LMS. At the second step, this approach was extended and 
realised as a PLMS supporting four learning phases of the SRL process (plan- 
search-learn-reflect). 

The use of the PLMS at workplace supports, on the one hand, curriculum-based 
learning allowing completing learning programmes and courses created by an 
organisation for its employees. On the other hand, it extends usual personnel 
development measures with SRL activities giving the employees an opportunity 
to specialise in the most important and interesting fields of knowledge, thus, 
developing their competencies and skills in an individualised manner addressing 
personal strengths and preferences. This assures gaining of obligatory knowledge 
and skills for completing specific tasks by the employees and supports personal 
development, thus removing borders between workplace and spare time learning. 

Moreover, while using the PLMS SRL competencies, such as goal setting, 
planning, time management, resources search, and self-evaluation, are trained. 
These skills can be applied by the learners not only in training context but also at 
the workplace helping to improve every-day working processes and achieving 
better results. Festo sees the need to support learners with the new technology 
and to develop SRL and media competence as key factors for successful working 
with the PLMS and the ROLE technologies. In this development process Festo 
keeps in mind that, in a business context, there are complex requirements and 
restrictions, for example, the contrasts between: 
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— Openness versus data security (e.g. the work council, the data protection officer 
and some others had very different opinions and views on the proposed work) 

— Companies’ targets versus individual targets 

— Implementation strategy: the “Revolution” (completely new technology) versus 
the “Evolution” (successive further development of existing tools) 


The conclusion, therefore, is that the current vision and deployment of a PLE 
towards an integrated PLMS implementation with predefined learning spaces on the 
technical side seems to have been warmly welcomed. Nonetheless, from the 
psycho-pedagogical perspective it remains essential that learners are able to learn 
in a self-regulated manner. 

The required technical improvements, therefore, have to be synchronised with 
the necessary individual development of specific SRL competences in order to meet 
these very real needs. The bigger pedagogical challenge, however, will be to 
promote the new approach to learners on a large scale. The evidence that this is 
possible is the fact that the LearningTube widget bundle has become an essential 
part of the Virtual Academy. 


Open Access This chapter is distributed under the terms of the Creative Commons Attribution 
Noncommercial License, which permits any noncommercial use, distribution, and reproduction in 
any medium, provided the original author(s) and source are credited. 
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Lessons Learned from the Development 
of the ROLE PLE Framework 


Sten Govaerts, Katrien Verbert, Evgeny Bogdanov, Erik Isaksson, 
Daniel Dahrendorf, Carsten Ullrich, Maren Scheffel, Sarah Léon Rojas, 
and Denis Gillet 


Abstract Within the ROLE European research project, an interoperability frame- 
work has been developed to support self-regulated learning and to enable learners 
and teachers to create personal learning environments (PLEs). This framework 
enables learners to assemble tools, services and resources together to create their 
own custom learning environment. This chapter discusses the overall architecture, 
the specific components of this architecture and the platforms in which we have 
integrated the ROLE framework. Additionally, we share the lessons learned from 
the design and development. Furthermore, we discuss our experience with the 
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ROLE development infrastructure and our collaboration within the ROLE devel- 
opment team and with several open-source projects. 


Keywords Development « Interoperability e Best practices e Lessons learned e 
Collaboration * Open source * Widgets e Web apps * Framework « Personal learning 
environment « PLE « Informal learning « Self-regulated learning * Social media 
platforms 


Introduction 


The proliferation of Web 2.0 technologies (e.g. wikis and social networks) has 
impacted the way users retrieve and use information and how they interact with 
each other (Maness 2006; Ullrich et al. 2008; Ashley et al. 2009). The abundance of 
Web-based tools and content creates many opportunities for Technology Enhanced 
Learning (TEL). 

The ROLE project aims to exploit Web-based tools and technologies to 
empower learners to construct their own personal learning environments (PLEs). 
The overall goal is to create flexible, Web-based, open technologies for the feder- 
ation and mash-up of learning services to empower the learner to build her own 
responsive learning environment. Responsiveness is defined as the ability to react to 
the learner needs—i.e. through recommendation, adaptation or visual analytics 
services that support the learner to be aware of and reflect upon her own learning 
process (Fruhmann et al. 2010). The project also targets critical transition stages of 
lifelong learning, e.g. due to shifts in learner interests or when leaving the university 
and entering a company. Chapter 1 elaborates more on the ROLE vision on PLEs. 

Learning management systems (LMSs) such as Moodle, CLIX and Blackboard 
primarily focus on distributing learning content, organising the learning process and 
serving as interface between learners and teachers. Dalsgaard (n.d.) notes that in 
LMSs generally different tools, such as discussion forums, file sharing, whiteboards 
and e-portfolios, are integrated in a single system that bundles all tools necessary to 
manage and run courses. In contrast to PLEs, LMSs place a strong emphasis on how 
to centralise and standardise the learning experience (Guo et al. 2010). Learning 
activities in an LMS-based course are organised within a centrally managed system, 
which is driven by the needs of the institution. On the other hand, a PLE takes a 
more natural and learner-centric approach and is characterised by the free-form use 
of a set of services and tools that are controlled and selected by individual learners. 

In recent years, research on mash-ups has been elaborated, for example widget 
mash-ups have been deployed at Graz University of Technology (Ebner and 
Taraghi 2010). In addition, researchers have focused on augmenting traditional 
LMSs with widgets to provide live-updating and flexible applications. Wilson 
et al. (2009) have implemented widget support for Moodle. Their big challenge is 
logging student activities with the widgets, as there is no communication between 
the widgets and the LMS. 
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The ROLE framework builds on this existing work, but incorporates additional 
core technologies such as inter-widget communication (IWC), automated user 
activity tracking, collaborative spaces and authentication and authorisation services 
to protect data. This is the basis to enable real-time communication between 
widgets and users, and to automate user activity tracking from tools and services. 
The analysis of such data and IWC provides the basis to develop responsive systems 
that can react to learner needs in a coordinated way. 

Within the time span of the ROLE project, a new Apache project, called RAVE, ' 
emerged with the aim to provide an extensible mash-up platform for using, inte- 
grating and hosting widgets with personalisation, collaboration and content inte- 
gration features. The features of Apache RAVE and the ROLE project are quite 
similar, as confirmed by recent research that has been applying RAVE in educa- 
tional contexts (Pierce et al. 2011; Chudnovskyy et al. 2012). Since the RAVE 
project started during the development of the ROLE framework, ROLE did not 
adopt Rave, but rather contributed components to the RAVE project (which is 
discussed in more detail in section “Contributing ROLE Software to Open-Source 
Projects’). 

This chapter presents the ROLE interoperability framework, which is a technical 
platform to assemble widgets within responsive open learning environments. The 
framework allows the assembly of widget bundles with communication channels, 
authentication and authorisation mechanisms and services for activity tracking and 
analysis. The framework ensures that the widgets have access to the necessary 
information to react to learner needs. Furthermore, the platforms, on which the 
interoperability framework has been integrated, are discussed and the lessons 
learned from the design and development of the framework components are 
presented, as well as on the technical collaboration within the ROLE project and 
with open-source projects. 

This chapter is organised as follows. First, the overall architecture of the 
framework is presented in section ‘The Interoperability Framework’, after which 
each component is discussed in more detail. Section “ROLE Platforms’ elaborates 
on the different platforms that integrate the ROLE infrastructure and the repository 
of widgets. Afterwards, the organisation of the ROLE developer community and 
our contributions to open-source projects are discussed. Finally, the achieved 
results are summarised and their dissemination is discussed. 


The Interoperability Framework 


The purpose of the ROLE interoperability framework is to support assembly of 
different widgets in responsive open learning environments. The architecture sup- 
ports communication between widgets, authentication and authorisation 


! Apache RAVE, http://rave.apache.org/ 
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mechanisms, services for activity tracking and analysis and widget spaces, which 
manage widgets, resources and users. All these services can be accessed via open 
and if possible standardised interfaces. These are necessary for third-party devel- 
opers who want to create applications based on ROLE technology. The next section 
details the overall architecture. 


The Architecture 


The ROLE architecture is illustrated in Fig. 1. IWC (see section “Inter-widget 
Communication’) is used and managed by spaces, but is also an autonomous 
component. It depends on JavaScript and the XMPP (Saint-Andre 2004a, b) 
protocol to provide a user-, community- and space-centred remote IWC. This 
allows developers to build powerful collaborative real-time learning tools and 
learners to assemble them easily in responsive open learning environments. 

Tracking of activities is done via the Contextualised Attention Metadata (CAM) 
framework (see section ‘Contextualised Attention Metadata’). An event-based 
schema was developed to model user behaviour in learning environments. Events 
are tracked and sent to either a central or container-specific repository. IWC is used 
to track such events. The data is stored and retrieved via an REST API. 

As the CAM service contains sensitive data, an authorisation and authentication 
framework has been developed to protect this data (see section ‘Authentication and 
Authorisation’). It is also needed for other ROLE services that handle personalised 
data. One of the main goals of this framework is to reduce the amount of user 
interaction by providing a Single Sign On (SSO) authentication mechanism. 

Finally, widget spaces (see section ‘Spaces’) allow learners and instructors to 
create portable collaborative learning environments. Spaces consist of learners, 
configurable services and sharable resources, within a learning context. The space 
features can be provided either by a (OpenSocial) container itself or by a special 
widget. Such an approach guarantees container independence. Furthermore, widget 
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Fig. 1 The ROLE architecture 
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spaces provide a simplified single point of access to the other background services 
via an extended OpenSocial API. 


Inter-widget Communication 


IWC enables event-based communication between widgets following the Publish— 
Subscribe communication pattern (Birman and Joseph 1987; Eugster et al. 2003). 
We employ both local inter-widget communication (LIWC) within a PLE and 
remote inter-widget communication (RIWC) among different users, computers 
and PLEs. 

LIWC is realised in the OpenApplication Event API (Isaksson and Palmer 2010, 
n.d.) using the HTML5 Web Messaging standard (Hickson 2011) available in most 
major browsers, including backwards compatibility for the Google Gadget PubSub 
mechanism. Instead of ‘hard-wiring’ widgets with each other (Sire et al. 2009), all 
widgets within a PLE are notified of all events and then decide autonomously to 
react accordingly. If the widget acts upon the received event, a receipt is sent back. 
Supporting containers that receive such a receipt can inform the user, e.g. by 
highlighting the tool that sent the receipt. The event payload format is designed 
for partial semantic interoperability, i.e. developers use a combination of 
established vocabularies in a simplified format with name-spaced properties 
(e.g. Dublin Core (DCMI Usage Board 2006)). In practice, this means that when 
an event is broadcasted, the originating widget does not indicate what receiving 
widgets should do (only the past action is specified, e.g. select). If the originating 
widget had to specify the intent (i.e. the desired future action), it would need to have 
buttons or menu items for every conceivable proposable action in every conceiv- 
able widget (e.g. add to portfolio, share with friends, search in Wikipedia). With 
events, we instead choose to split the job: events should be broadcasted for as many 
user actions as possible within each widget, without concern for what receiving 
widgets ought to do, and receiving widgets provide the affordances (e.g. buttons) 
for their own proposed further actions. 

RIWC enables communication among widgets in different browsers and on 
different machines in order to foster real-time remote communication and collabo- 
ration functionality. RIWC is realised with the Extensible Messaging and Presence 
Protocol (XMPP) (Saint-Andre 2004a, b), an open standard for real-time communi- 
cation. The power of XMPP lies in its built-in federation capabilities and extensi- 
bility through XMPP Extension Protocols (XEPs), such as for Publish/Subscribe 
(Millard et al. 2010) and Multi-User Chat (Saint-Andre 2008) as applied in respon- 
sive and collaborative learning scenarios (Friedrich et al. 2011). Since no 
JavaScript XMPP library with PubSub support was available, ROLE extended the 
dojo XMPP library by a set of common PubSub operations. Users can discover 
nodes, retrieve subscriptions, create, configure and delete nodes, subscribe and 
unsubscribe nodes and publish/receive IWC events in an XML-based payload 
format across a federated network of XMPP servers. However, current libraries 
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using XMPP over BOSH (Paterson et al. 2010) are not applicable in public 
containers such as iGoogle due to cross-domain issues. Furthermore, they are rather 
unstable and unreliable (Friedrich et al. 2011). Our experiments showed that the 
upcoming Web Socket API (Hickson 2009) for XMPP (Moffit and Cestari 2010) 
outperforms BOSH with considerable performance and stability improvements and 
availability in all containers. 

IWC enables more responsive, collaborative environments with real-time noti- 
fications and richer user experience, although attention to usability is required 
(Isaksson and Palmer 2010). 


Contextualised Attention Metadata 


Tracking of user interactions with widgets is an essential part to enable responsive- 
ness in learning environments. User interaction data can be used for data analysis 
and the computation of personal, social and contextual information about users and 
applications. Additionally, such data of the actual usage of ROLE services in real- 
world settings was used to evaluate the framework. 

A variety of attention metadata formats exist. These formats differ in scope, 
expressiveness, scalability and context awareness. Butoianu et al. (2010) provide a 
survey of the following formats: TaskTracer (Dragunov et al. 2005), Swish (Oliver 
et al. 2006), CAM (Wolpers et al. 2006), the User Interaction Context Model 
(UICO) (Rath et al. 2009), the Context Modelling Language (CML) (Henricksen 
and Indulska 2006) and WildCAT (David and Ledoux 2005). TaskTracer and Swish 
are least flexible and expressive. UICO, CML and WildCAT are very expressive; 
however, the available frameworks using the formats do not scale well and some are 
focused on a specific application (Butoianu et al. 2010). On the other hand, CAM 
supports scalability and context awareness very well, but is less expressive. Other 
examples are the ActivityStreams specification” and the Experience API (Glahn 
2013). The latter took inspiration from ActivityStreams and only became available 
in the last years of the ROLE project. ActivityStreams are less focused on contex- 
tual information. In ROLE, CAM is used because of scalability and context 
awareness. 

The CAM schema (Schmitz et al. 2011) can be used to describe computer- 
related activities of one or several users—i.e. which objects attract user attention, 
which actions users perform and what the user contexts are. CAM was developed to 
describe as many types of attention metadata as possible. Therefore, CAM records 
of a user cannot merely describe user foci of attention, but rather her entire 
computer usage behaviour. Collections of CAM records can be exploited for 
generating diverse kinds of profiles like user profiles and object profiles (item 
profiles). CAM records represent user—computer-related foci of attention and 
actions and thus can instantly constitute profiles of individual usage behaviour. 


2 The ActivityStreams specification, http://activitystrea.ms/specs/json/1.0/ 
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CAM records of different users can be exploited for generating attention and usage- 
based object profiles. 

CAM records can be used to support self-reflection. For instance, visualisation 
widgets can support a user to recapitulate what she did and generate a picture of her 
competences. Furthermore, statistical metrics can be employed to aggregate and 
evaluate CAM records of different users. By this, general trends, for instance in 
computer usage, data consumption and communication behaviour, can be detected. 
Aggregated CAM records entail information on the behaviour of average users and 
on the behaviour of user groups. They also entail usage information on data objects, 
such as how often a certain object was used, and by which kinds of users and in 
what contexts it was used. In addition, they can reveal in which respect a user 
deviates from the average, whether her behaviour conforms to general trends or 
not, etc. 

The CAM schema has been developed to provide a unified schema for monitor- 
ing data across system boundaries (Wolpers et al. 2007; Schmitz et al. 2011). The 
CAM schema has been transformed from a once user-centred version to an event- 
based version (see Fig. 2) that is better suited for evaluating and analysing user 
observations over time.” 


3 Information about the schema is available at https://sites.google.com/site/camschema/ and the 
CAM API: http://sourceforge.net/projects/camapi/ 
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One major goal of ROLE is to provide personalisation, recommendation and 
self-reflection mechanisms. To achieve this, users can be monitored while 
interacting with their learning environment. The collected CAM information is 
used to generate different patterns and statistics, such as discovering learning trends 
and detecting what is currently happening in the learning environment. For an easy 
integration of the CAM monitoring into different learning environments, the moni- 
toring architecture was divided into a client and a server component. The client 
component can be considered as a data collection element, responsible for accumu- 
lating and transforming the information into CAM, while the server component is 
responsible for the persistence management and data access control. 

Figure 3 shows the CAM architecture applied in ROLE and how it works 
together with other ROLE components. The picture shows a platform, which uses 
ROLE technology by integrating ROLE widgets into their learning environment 


Platform (e.g. SJTU, FESTO, ...) 
Container (e.g. Liferay, CLIX) 


Application Server (e.g. Tomcat) 


a a — > 
| CAM -n oars 

ar Widget 1 Widget 2 Widget storelquary W 
l External CAM DB 


CAM WebService 


Platform (e.g. SJTU, FESTO, ...) 
Container (e.g. Liferay, CLIX) 


Application Server (e.g. Tomcat) 


= —— — 

| CAM Œ «<—___> | 

Widget 1| > | Widget 2 Widget storeiquery i 
External CAM DB 


a CAM WebService 


Platform (e.g. SJTU, FESTO, ...) 
Container (e.g. Liferay, CLIX) 


Application Server (e.g. Tomcat) 


= = 
Widget 1 Widget 2 z œŒ Aae e 


External CAM DB 
seme CAM WebService 


Fig. 3 The CAM architecture used in ROLE 


Lessons Learned from the Development of the ROLE PLE Framework 193 


~ CAM Activity Meter - 7 x 


privacy setting: monitoring on ~ 


E you(tnYmg-AlS_atZYMxDko-Pg) 


activity perUser -| @ for today aK iJ 


refresh 


Fig. 4 Screenshot of the ROLE CAM Monitoring widget 


(container). Furthermore, the picture illustrates that the CAM monitoring compo- 
nents must not be integrated into the platform since they are running on an external 
application server. First, a user performs an action on a widget (step 1), e.g. clicking 
a button. Since this widget supports IWC (see section ‘Inter-widget Communi- 
cation’) this action causes an event, which is published via the OpenApp mecha- 
nism (Isaksson and Palmer 2010; Isaksson and Palmer n.d.). Thus the event is 
broadcasted (step 2) and can be received by all other widgets in the user’s learning 
environment (local IWC, see section ‘Inter-widget Communication’). Each receiv- 
ing widget can process and react on the event (step 3). The CAM widget can thus 
receive all events sent through IWC. Afterwards the CAM widget identifies the 
event and forwards all required information to the CAM Web service, which is 
responsible for the CAM mapping and persistence. To offer the user an overview 
over the activities in the learning space, it contains a graph where past events from 
the users of the space can be displayed using different configurations, i.e. one or 
several users and several dates (see Fig. 4). As already mentioned, the picture 
illustrates that the Web service is not integrated into the platform, but is a stand- 
alone service that can be installed for a specific platform. 

In addition to storage, the CAM Web service provides a query method, which 
consumes an arbitrary SQL select statement and returns the results in the JSON 
format. Using arbitrary SQL select statements ensures maximum freedom for the 
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developers to experiment with the data. Access to the monitoring data causes 
privacy issues. These issues have not been completely resolved in the ROLE 
project. Some steps have been taken though. For instance, data privacy can be 
supported through a stand-alone installation of the CAM Web service, e.g. by 
installing the CAM service in the intranet of an institution. The CAM widget also 
allows users to disable tracking (see Fig. 4). Additionally, the CAM Edit and Share 
widget* (see Fig. 5) allows the user to filter and export her CAM data from a specific 
space into an SQL file. This widget provides the user with full access and larger 
control over her CAM records, and she can share her events with others or analyse 
them herself. Finally, since the Web methods for querying and storing CAM are 
publicly available, the CAM service requires password-based authentication. 

From our experience developing the CAM service and using it in real-world 
settings various new insights have been gained. One of the main benefits of the 
approach is that developers do not have to specifically write code to track user 
events, as the CAM widget will collect the IWC events automatically. On the other 
hand, this can also limit the data collection since not all important events might be 
requiring IWC. This tracking method also allows developers to be agnostic about 
the CAM schema and the implementation of the CAM Web service, since they do 
just need to send out IWC events. One of the strengths of the OpenApp IWC is the 
openness of its data format, which has no mandatory fields and enables developers 
to transmit any kind of JSON data from one widget to another. This freedom makes 
the mapping of the OpenApp events into CAM harder as such mapping cannot rely 
on certain elements to be present. Defining a subset of fixed mandatory IWC fields 
and fixing taxonomies of event types would make the CAM mapping more easy and 
robust. 

Another flexibility issue is due to the high abstraction level of the CAM schema 
to allow all kind of events to be stored. This can generate a large number of different 
CAM event mappings, which can make it harder to analyse and compare the CAM 
records. The different IWC events stored information in different CAM fields, 
making the data analysis more complex without knowing the details of the intrinsic 
mapping. The ActivityStreams specification provides an extensible, common 
vocabulary list of actions that would aid this and when applied properly could 
provide more portable data (Vozniuk et al. 2013). 

Currently, user activities are only tracked when the CAM widget is added to the 
learning environment. Hence, this enables users to have full control over where 
their activities are tracked, but also causes that nothing is recorded if the user 
forgets to add the CAM widget. This problem could be circumvented by integrating 
CAM monitoring directly into the platform. 


4 http://www.role-widgetstore.eu/tool/cam-edit-and-share-widget 
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Fig. 5 Screenshot of the 
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Spaces 


In the ROLE framework, a PLE can consist of various spaces. A space is an abstract 
concept that materialises the user’s context and aggregates people, resources, 
applications and other subspaces. All these artefacts belong to the same activity 
that a person (or a group of them) is working on to achieve a common goal. This 
common goal is the purpose to create the space. Various people can participate in a 
space and might have different access rights and roles within this space, where they 
share resources and applications that they need to achieve their goal. A space might 
have subspaces that help hierarchical organisation of resources, applications and 
people. A space can be seen as a PLE unit. On the one hand, a space is a way for 
users to give shape to their PLEs by aggregating information. On the other hand, a 
space allows users to share their PLEs with others by inviting them to collaborate. 

The space concept exists in all ROLE platforms (see section ‘ROLE Platforms’): 
the ROLE SDK (see section ‘ROLE SDK’), Graasp” (Bogdanov et al. 2012a) (see 
section “Graasp’), the OpenSocial Moodle plugin (Bogdanov et al. 2012b) (see 
section ‘OpenSocial Moodle Plugin’) and the Widget Store (see section “The ROLE 
Widget Store’). Every platform internally implements this concept in a different 
way. In order to allow widgets to use the information about the space and to enable 
integration and portability of spaces between the platforms and beyond, we applied 
two approaches: Linked Data and OpenSocial. 

For the first approach, we created a space ontology for Linked Data. Linked Data 
provides a very powerful and extensible way of describing data in a machine- 
understandable way. It targets the discovery and integration of data originating 
from different sources. Due to its design, it has limitations. First, Linked Data and 
SPARQL require a rather steep learning curve, which is a disadvantage compared 
to simple RESTful APIs that are used by many Web developers. The second 
limitation is the performance. Since the data is located on different servers, many 
HTTP requests have to be issued to retrieve the complete data. Moreover, the 
SPARQL engine requires traversal of a graph, which is much slower than retrieving 
data from a relational database. The authors Health and Bizer (n.d.) (see section 6.3) 
foresee the use of data crawling for real-time Linked Data applications, rather than 
on-the-fly URI dereferencing. 

The alternative approach we used is OpenSocial. The OpenSocial specification 
consists of three main parts. The first part describes the widget standard. The second 
part standardises the model for social network elements (i.e. Person, App and 
Document) and relations between them. The third part standardises a set of com- 
mon REST and JavaScript APIs to retrieve data from a social platform. Since the 
space concept did not exist in OpenSocial, we introduced it into the OpenSocial 
specification. The OpenSocial Space extension standardises the space model 
(namely a list of fields that a space can contain), and the REST and JavaScript 


5 Graasp, http://graasp.epfi.ch 
© Linked Data API, http://code.google.com/p/linked-data-api/wiki/API_Rationale 
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APIs to work with spaces (Bogdanov et al. 2011). Through this extension, widgets 
can retrieve information about their containing space and its content via OpenSocial 
APIs. For example, the Person API can be used to retrieve all the members of the 
space. The widget can retrieve a list of the resources and widgets available in the 
space via the Document and App APIs, respectively. 

The main disadvantage of OpenSocial is that the social model cannot be easily 
and arbitrarily extended as with Linked Data. The new extensions require going 
through the process of standardisation, which can be quite cumbersome. On a 
positive note, OpenSocial provides easy-to-use REST APIs with a JSON-based 
data representation. The data format and APIs are standardised, which enables 
interoperability when data is accessed and processed. OpenSocial does not target 
data discovery (as Linked Data) but rather data retrieval and exchange. Since the 
data is often centralised in one institution, it is very fast to retrieve the data 
compared to the SPARQL engine. 


Authentication and Authorisation 


CAM contains sensitive and personal data protected by law. Additionally, users 
might prefer to keep the content of their spaces private. The data access has to be 
trusted and allowed by the users. The data communication occurs at two different 
levels: service-to-service and widget-to-service communication. 

Service-to-service communication can occur when for example a recommen- 
dation service requires CAM relevance feedback on resources. Data can be trans- 
ferred across institutions and countries with different laws. Thus, we decided to 
leave the decision of service-to-service authentication and authorisation (A&A) up 
to the service developers. 

Widget-to-service communication occurs when for example a self-reflection 
widget wants to query the CAM service. This has been implemented as follows. 
A user is authenticated as being the owner of a particular personal space. The user 
first authenticates as being the owner of a separate identity, to which the personal 
space is linked (or a new personal space will be created), which thereby implies that 
the user is the authenticated owner of the space. The personal space then functions 
as the identity of the user.’ Authentication is typically done via OpenID, which is 
the standard for decentralised authentication on which we have focused, but other 
protocols may be supported as well (one test bed, at Uppsala University, has 
implemented support for CAS while also keeping support for OpenID 
authentication). 

Delegated authorisation is used by widgets that access collaborative and per- 
sonal spaces. Furthermore, such delegated authorisation may also be used by third- 


7 The identifier of the personal space, i.e. a URI, is used as the identifier of the user, cf. WebID (see 
http://www.w3.org/wiki/WebID) 
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party services. The standard for delegated authorisation that we have focused on, 
i.e. OAuth, lets the currently authenticated user choose whether to authorise the 
widget or service. After authorisation, the service is provided with a token granting 
access to spaces on the user’s behalf. For widgets, the token is managed by the 
OpenSocial widget container, which allows widgets to perform requests through the 
engine’s OAuth proxy that first applies OAuth and then forwards the request. 

Currently, OAuth endpoints for OpenSocial widgets must be hard coded in the 
widget’s source XML. Therefore, spaces implement a rewriting of the XML so that 
the proper endpoints are included. Otherwise, it would be necessary for widget 
developers to maintain separate XML files of their widgets for every server where 
the widget is deployed. Widgets using the rewritten XML files, however, can be 
added to any widget container, such as Liferay? (Yuan 2009), while still 
maintaining the connection to their respective spaces. 


ROLE Platforms 


The ROLE interoperability framework has been integrated in various platforms. 
This section describes these platforms where users create and use their PLEs and 
search for widgets. 


The ROLE SDK 


The ROLE SDK is a collection of software and tools, which allows trying out 
ROLE technology and developing new widgets for mash-up PLEs. In total, ten 
versions of the ROLE SDK were released, each one packaging the implementation 
outcomes of one milestone of an iterative development process (see section ‘The 
ROLE Developer Community’). 

The central core of the SDK is the reference implementation of a sample PLE, 
which allows using ROLE technology in practice and developing new widgets at 
the same time. Within the ROLE SDK, a learning space functions as a collaborative 
context for learning, consisting of a bundle of widgets, along with a list of 
participants. Widgets can interact with other widgets, and participants can interact 
with other participants, by using widgets and built-in features of the ROLE SDK 
(e.g. chat functionality). 

A personal space is defined as a personal context that consists of a person’s user 
model. One representation of the user model is a user profile; another representa- 
tion, also based on the user model, is a bundle of (personally chosen) widgets. In the 


s Liferay, http://www.liferay.com/ 
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ROLE SDK, the learning space and personal space are combined in one user 
interface. 

Additionally, there is a social context. The social context offers access to the 
communities of which the user is a member. The specific community that is 
accessed typically depends on the website where the widget is currently being 
used, which may be a different website than that of the PLE. For instance, if a 
widget is being used on a social networking website, the community would be that 
of the website. Collaborative contexts (i.e. learning spaces) can transcend social 
contexts. A widget may be part of a learning space, and at the same time be used on 
a social network for inviting people from that network to participate in the space. 
OpenSocial standardises APIs for access to what is defined as the social 
context here. 

Furthermore, the concept of activities was introduced. Activities can be defined 
as purposes for which the user structures her learning context and assembles 
widgets. In the ROLE SDK, activities are displayed as one group of visible widgets 
at a time. In the GUI, activities can also be tabs or pages. However in the ROLE 
SDK, the additional semantics that the term ‘activities’ offers is covered. The term 
hints to the user that the groupings should be used for focusing on one activity at a 
time (such as training English vocabulary or searching for Web resources), using 
the tools that are appropriate to that activity, without being distracted by what is 
unrelated to the activity. 

As mentioned before, the ROLE SDK relies on the concept of spaces. While a 
space is, at its most basic level, simply a bundle of resources such as widgets, there 
are several aspects that contribute to its usefulness as the basis for a PLE: 


e Aggregation: Widgets (or more generically, tools) are bundled with any other 
kind of resources that contribute to the space’s goals. The model enables a very 
flexible use of spaces, without requiring modifications in the model or its 
implementation. 

e Contextualisation: A space forms a context for its contained resources. Widgets 
can be made context aware, and are then able to interact with the space and its 
resources. In addition to being in the context of a space, widgets can be 
contextualised further by being given configuration that is specific to their 
instantiation within the space. 

e Participation: People can join a space, which means that they become members 
of that space. Members are notified of the presence of other members, and can 
interact with them both asynchronously and synchronously. 

¢ Personalisation: Spaces offer a level of customisability, so that users are able to 
personalise the environment according to their needs. 


These design requirements are realised within the ROLE SDK. Being a sample 
implementation of a PLE platform, the space user interface (see Fig. 6) is a Web 
application composed of four parts. First and on top, the header element is 
implemented as a top-aligned bar. It provides elements for controlling the Web 
application as a whole, such as signing in and out, and navigating to other parts of 
the application. Secondly, the sidebar element is a narrow, fixed-width section, 
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Fig. 7 The ROLE Widget Store, embedded in the ROLE SDK 


running along the left side. It provides elements for controlling the space, such as 
switching between activities, and adding widgets. 

Thirdly, the container element comprises the central area of the user interface, 
not covered by the other parts. This is where the main content is located, typically 
the space’s widgets. The container can also be used as an embedded browser, which 
is how the ROLE Widget Store (see section “The ROLE Widget Store’) is inte- 
grated (see Fig. 7). 

The fourth and last element is the dashboard, a bottom-aligned bar when 
collapsed. Expanding the dashboard displays the widgets on the personal space 
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Fig. 8 The personal space, inside the expanded dashboard 
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Fig. 9 The personal space, being accessed on a third-party website 


right above the container, as illustrated in Fig. 8. The dashboard provides access to 
the user’s personal space, which is a space that is private to the authenticated user. It 
is available from any other space (and other pages of the Web application) as well 
as from third-party websites via a bookmarklet (cf. Fig. 9). 

In the case that the space itself is embedded (e.g. on the course page of an LMS), 
it is intended that the parts can be hidden or moved (e.g. the Header), because their 
functionality (e.g. sign in) can be already provided by the LMS or to adhere to 
another design. 
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Fig. 10 A shared contextual space created in Graasp that integrates resources gathered from the 
Cloud, such as YouTube videos, SlideShare presentations, OpenSocial Widgets, Web pages or 
PDF documents with previews 


Graasp 


Graasp” (Bogdanov et al. 2012a) is a social media platform for collaborative 
learning and knowledge management (see Fig. 10). Graasp implements the 
OpenSocial space specification (see section “Spaces’), which enables the creation 
of spaces shared between people belonging to different communities and networks. 
Embedded shared resources are gathered across institutional and corporate bound- 
aries. Unlike dominant social media, Graasp enables a fine definition of the 
audience, as well as the associated rights and roles to ensure trust and privacy 
enforcement. In Graasp, people map their personal and shared projects, interests 
and activities into public or private contextual spaces integrating invited members, 
relevant resources and necessary apps which can be tagged and rated. Any space or 
resource in Graasp integrates its own discussion thread to enable contextual inter- 
action. Graasp allows learners to construct and manage their own PLEs. Users can 
create a PLE for each learning objective, populate it with various resources and 
tools, personalise it and share it with others (Bogdanov et al. 2012a). 


° Graasp, http://graasp.epfi.ch 
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The space concept is at the core of Graasp. A space can represent a PLE and can 
contain four types of entities: people, resources aggregated and used within the 
space, apps added to the space to extend its functionality and subspaces to organise 
the space content in a hierarchical structure. Graasp enables users to manage their 
spaces. 

The resources and apps can be aggregated into a Graasp space from both local 
and remote locations. First, users can easily drag and drop files directly from their 
desktops into their spaces. Second, remote resources from the Cloud can be easily 
aggregated via an aggregation mechanism called GraaspIt!. Whenever a user 
encounters an interesting page, she can simply click on the GraaspIt! bookmarklet 
in the browser and the resource will be added into a space (Gillet and Bogdanov 
2012). These collected resources can be aggregated as URLs, embed tags or Web 
page screenshots. In addition to resources, Graasp allows users to aggregate widgets 
into their spaces. Currently, the OpenSocial widget standard is supported, though 
other standards (e.g. W3C widgets) can be added in a similar way. Such widgets 
either can be added manually or can be aggregated from existing widget reposito- 
ries. For example, when the user is browsing through widgets in a widget repository 
(e.g. the ROLE Widget Store, see section ‘The ROLE Widget Store’), a widget of 
interest can be added by just clicking on GraaspIt!. The second way is to add 
widgets from the ROLE Widget Store by exploiting the widget repository search 
mechanism provided within Graasp. 

Once a space is created and populated in Graasp, the core part of the interface 
(see Fig. 10) enables users to interact with the aggregated content and can be further 
personalised with the concept of functional skins (Bogdanov et al. 2011). A 
functional skin is a client-side plugin for a space that can retrieve space data via 
the OpenSocial APIs and provides users with visual and functional features differ- 
ent from Graasp and tailored to specific needs. For example, in addition to the 
standard view of Fig. 10, Graasp offers two built-in functional skins: the resource 
view and the app view. The resource view displays a list of all resources that exist in 
a space and provides download links and presents resource previews. The app view 
displays all widget instances from a space as a visual mash-up. In this view, widgets 
can be resized and their order can be modified through drag and drop. The 
possibilities to personalise the space are extensive and through functional skins 
the users can further adapt their spaces for their own professional/personal tasks. 

Graasp implements several mechanisms to share and exchange spaces and 
widget bundles (sets of widgets combined together for a specific purpose). An 
app bundle can be extracted from the existing space and exported as an OMDL 
file.'° The OMDL file can be imported into another platform (or reused in Graasp) 
or shared at the ROLE Widget Store. Additionally, a space created in Graasp can be 
shared with other people and with other platforms. The space can be extracted from 
Graasp as a secret URL. This URL can be given to other users and allows them to 
collaborate anonymously. Alternatively, the space can also be embedded into 


10 Open Mashup Description Language, http://omdl.org 
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Fig. 11 A teacher creates a space with widgets for a course 


another Web platform. Chapter 5 provides more information on how Graasp was 
used and evaluated for formal learning. 


OpenSocial Moodle Plugin 


The OpenSocial Moodle plugin (Bogdanov et al. 2012b) enables the use of 
OpenSocial widgets within the Moodle LMS'' to create PLEs. By providing 
support for PLEs in an existing LMS, the disruption often caused by providing 
users with completely new environments decreases. By integrating PLE features in 
an existing LMS, users can still continue to use the features of the familiar LMS, but 
can personalise their learning environment with widgets. 

The OpenSocial plugin for Moodle exists in two variations. The first version 
adds a new module to Moodle, which is displayed in the central area in the Moodle 
UI.'* The module allows a teacher to add a ‘widget space’ to the Moodle course 
page, specify a set of widgets and choose the widget layout on the Moodle page (see 
Fig. 11). After this configuration, students can work with several widgets simulta- 
neously (see Fig. 12) in the Moodle course. 


1 Moodle, http://www.moodle.org 
12 OpenSocial Moodle module, https://github.com/vohtaski/shindig-moodle-mod 
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Fig. 12 OpenSocial widgets displayed within Moodle 


The second version”? of the OpenSocial plugin adds a new block in the right 
column of the Moodle UI. With this Moodle block, the teacher can add widgets to 
the right column of existing Moodle pages. Both versions of the Moodle plugin 
make use of the Apache Shindig engine,“ which provides an open-source imple- 
mentation of the OpenSocial specification, to render and manage widgets. 

One of the main benefits of these plugins is that they enable teachers to easily 
extend Moodle with new features and services provided by widgets. Consequently, 
once the OpenSocial plugin is installed in Moodle, a teacher can append the 
required functionality herself, without the intervention of system administrators. 
The plugin enables the flexibility of selecting the resources and tools required for a 
specific course. Additionally, the plugins enable reuse of existing educational 
resources and tools. Furthermore, teachers and students can continue to operate in 
the learning environments they are familiar with but gain the mash-up features of 
PLEs. Naturally, the components of the ROLE architecture are compatible with the 
Moodle plugin. For instance, IWC and CAM are fully operational in the Moodle 
plugin. By extending widely used LMS with PLE features, we aim to achieve a 
faster adoption of the PLE paradigm among institutions. Further details on how this 


'3 OpenSocial Moodle block, https://github.com/vohtaski/shindig-moodle-block 
14 Apache Shindig, http://shindig.apache.org 
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Moodle plugin was put to use for formal learning and evaluated are available in 
Chap. 4. 


The ROLE Widget Store 


The ROLE Widget Store allows users to search and browse for widgets and 
compilations of them. The store addresses the issue of categorisation, browsing, 
searching and recommending by providing various widgets categorised based on 
functionality, learning phases and learning domains. Further, the Widget Store 
enables sharing of platform-independent PLE and templates composed of learning 
tools and artefacts (or the so-called Widget Bundles). Via these mechanisms, the 
Widget Store fosters the development of a community of practice to exchange 
learning tools. Regarding the widget bundles, the store provides features to apply 
and share bundles across different learning platforms. This section further discusses 
different recommendation strategies and the interfaces that enable interoperability 
are specified (LMS/PLE system integration). Figure 13 presents an overview of the 
Widget Store architecture. 

The main focus of the store is to provide a catalogue of widgets by supporting 
two commonly used widget specifications: the W3C widget specification (Caceres 
n.d.) and the OpenSocial specification (Mitchell-Wong et al. 2007). Developers can 
post either their self-developed widgets or widgets based on licenses which allow 
further distribution. Where possible metadata are automatically extracted from the 
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Fig. 13 Components of the ROLE Widget Store 
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widgets. Widget bundles are compilations of widgets, which are created to share 
good practices of widget use in learning environments. They are intended as a fast 
and simple way to provide learners with tools, services, content and a detailed 
description of how to use these to complete a specific learning task. Learners can 
select several tools from the store to create their learning environment. Additional 
references to learning resources can be added. For each tool and resource, learners 
are able to add learning activities in order to describe what should be done using the 
tool or learning resource. Once such a bundle is created by a learner, she can share it 
with the community. Such bundles can be reused by teachers and learners for their 
learning environments. In order to support learners in selecting applications for 
their PLEs three different categorisations are provided. 


¢ Tool categories are derived from the Psycho-Pedagogical Integration Model 
(PPIM) (Fruhmann et al. 2010) (more information on PPIM is also available in 
Chap. 2), so users can select widgets supporting different learning phases. 

¢ Tool functionalities represent features of widgets (e.g. text editing, video chat) 
and are based on an ontology developed in ROLE. 

¢ Learning domains describe, if possible, the domain of the tools by providing 
semantic tags using DBpedia." 


The categorisation of bundles differs from the tool categorisation. A bundle can 
be designed to cover several phases of the PPIM model and thus refers to several 
tool categories. The approach of the Widget Store is that a bundle automatically 
inherits functionalities of tools it contains and can be tagged manually by learning 
domains from the DBpedia. 

To provide an interface for external systems, the ROLE Widget Store offers an 
SPARQL endpoint which allows retrieval and insertion of the data of the Widget 
Store based on a standardised interface. Furthermore, different formats (Turtle, 
RDF/XML and JSON) are supported so that developers can choose their preferred 
data format. Another possibility for PLE platforms to integrate the store is to embed 
the store in the learning environment. The embedded version provides a simplified 
user interface and offers buttons that allow users to directly choose widgets to 
assemble their PLE. The store is connected to other ROLE components in the 
following ways: 


e Graasp queries the store to provide a catalogue of widgets enabling easy 
integration of widgets in Graasp spaces. 

e The ROLE SDK embeds the store and uses the embedding features to add 
widgets to the ROLE spaces. 

e The ROLE Pedagogical Recommender (Nussbaumer et al. 2012) queries the 
store to provide recommendations based on the ontology of learning activities 
and the store categorisations. 


= DBPedia, http://dbpedia.org/ 
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* The ROLE Requirement Bazaar'® (Renzel et al. 2013) uses the data to support 
the requirements elicitation and negotiation process being part of the ROLE 
Social Requirements Engineering approach. The ROLE Requirement Bazaar is a 
collaborative social platform where users can illicit their needs and wishes to 
developers who can extract requirements for future implementation. 


Widgets and Tools 


One major problem regarding the adoption of ROLE in new test beds and increas- 
ing the number of users was the limited number of widgets that were available. 

One approach to overcome this problem is to enable a very simple transforma- 
tion of existing Web resources into widgets (Ullrich et al. 2013). This transforma- 
tion can be done by developers through the usage of widget templates as well as by 
non-technical people with the help of an authoring tool. Both solutions support the 
same ROLE technologies, namely the capturing of interactions via CAM and the 
possibility to rate the widgets. Interactions are captured on a very generic level: 
basically, whenever a student uses a Web application integrated into a widget for a 
period longer than five seconds, then the widget sends out CAM event of the type 
‘used’. Of course, a developer can refine the interactions, when required. 

The ‘widgetisation’ of a Web application is simplified through the usage of a 
template and through JavaScript libraries that can be included in (existing) widgets. 
The template defines a widget that embeds the Web application via the iframe 
HTML element. This has the advantage that the original Web application does not 
need to be modified. In case, a widget of the Web application already exists, the 
capturing of interactions via ROLE can be enabled by the inclusion of the 
JavaScript library. This library uses IWC to send out the captured interactions, 
which can be made persistent on the CAM service via the CAM Monitor widget as 
described in section “Contextualised Attention Metadata’. 

The proposed approach has been implemented and a widget template is available 
in which the widget developers have to add the link to the Web page they want to 
integrate. To extend an existing widget, one has to include several lines of 
JavaScript code. The generation can be automated by using a set of shell scripts. 
The scripts take a list of URIs as input and generate widgets for the URIs. This 
reduces the authoring time to less than a minute. In summary, while this solution 
works very well for advanced software developers, it is still too complex for the 
average user. As an example, the code to extend an existing widget with ROLE 
technologies looks as follows: 


16 ROLE Requirements Bazaar, http://role-is.dbis.rwth-aachen.de:9090/BazaarFrontend/index. 
html 
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//Load two libraries for allowing the user to rate this gadget//and for 
capturing interactions in CAM format $.getScript("http://widgets. 
onlinesjtu.com/gadgets/libs/rating.js", 


function () { 
$.getScript ( 
"http: //widgets.onlinesjtu.com/gadgets/libs/interactioncapture. 
js", 
function () { 


var rating = new ROLE_module.rating 
("#importedGadget") ; 
var interactioncapture = 


new ROLE_module.interactioncapture 
("#importedGadget") ; 
}) 
} 
i 

The first lines load the libraries. The functionality is activated by creating the 
appropriate objects. In the example, #importedGadget specifies the HTML element 
to which the interaction capturing and rating functionality should be attached 
(typically a div element, which is the parent of the iframe element). 

In addition to the simplification of the usage of the libraries, the Shanghai Jiao 
Tong University (SJTU) created an authoring widget that allows teachers without 
Web development expertise to generate widgets from existing Web resources. The 
authoring widget asks users to input the URI of the Web application and add some 
metadata. Then, the authoring tool generates and uploads the widget to a server. 
Through an integration with the OpenSocial Moodle module (see section 
‘OpenSocial Moodle Plugin’) users can create widgets without having to leave 
the learning environment. 

Thanks to these tools, SJTU was able to create several hundreds of very domain- 
specific widgets for ROLE. Additional details on the SJTU test bed are presented in 
Chap. 4. 


ROLE and Open-Source Developer Communities 


All technical partners of the ROLE project have been collaborating successfully to 
create the ROLE framework. To foster this collaboration, various support mecha- 
nisms were set up, consisting of management structures, sub-projects, development 
software and developer meetings. This section elaborates on the developer collabo- 
ration within the ROLE project and with open-source projects to disseminate ROLE 
technologies. 
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The ROLE Developer Community 


Technical cluster structure: To enable the assessment of requirements, exploration 
of technologies, creation of early prototypes and their evaluation, the development 
process was split up in consecutive sub-projects, each having its specific goals and 
deadlines. In total there were five of such projects: the Christmas project (ended on 
Christmas 2010), the Easter project (ended on Easter 2011), the Stonehenge project 
(ended on December 22, 2011), the Gunpowder project (ended on January 
31, 2012) and the Shori project (ended on January 31, 2013). By defining use 
cases and goals for each project, the requirements and planning were defined. These 
projects also allowed easier planning of evaluations. The longer projects (i.e. the 
Gunpowder and Shori project) had a more elaborate planning phase and manage- 
ment methodology. For the Gunpowder and Shori project we aimed to apply the 
SCRUM (Schwaber 2004) and Kanban (Ladas 2009) methodology. But due to the 
large, geographically dispersed team from different organisations, we opted for an 
adapted version of SCRUM combined with Kanban, where one or two persons 
would manage the development process and report progress to the ROLE general 
assembly. The requirements and goals for the projects were often laid out in a face- 
to-face developer meeting or developer camps (see below) and follow-up virtual 
meetings. The project managers would then plan milestones (or sprints) often based 
on evaluation deadlines and showcases at conferences. Certain topics had smaller 
teams working on it in task forces, e.g. assessing a solution for authentication or 
CAM. Bi-weekly technical virtual meetings were organised to discuss progress, and 
to decide on technology and architecture choices. This setup allowed all developers 
to work on their own tasks and be involved in the decision making, but also get an 
overview of the current project status. Furthermore, it allowed the project managers 
to follow up the progress and react quickly where needed. This approach was 
received positive by developers, managers and general assembly. Next to this 
methodology, the development was also assisted by software. 

Development software: To support the developers and the management, several 
software packages were set up. To provide access to and version control our source 
code we experimented with Git!’ and Subversion (SVN).' Initially, Git on 
GitHub”? was used, but at that time GitHub did not fulfil the requirements of the 
project. To reduce the complexity, the source code was migrated to Subversion.”° 
At the end of the project, the source code was migrated again to GitHub,”' since 
GitHub has a more flexible scheme where any external developer can reuse the 
source code without any intervention from the repository owner. Whereas 


17 Git, http://git-scm.com/ 

18 Subversion, http://subversion.tigris.org/ 

19 GitHub, https://github.com/ 

20 The Subversion repository is available on Sourceforge at http://sourceforge.net/projects/role- 
project/ 

?! The ROLE GitHub repository is available at https://github.com/organizations/ROLE 
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Sourceforge still requires management by a ROLE partner. To manage the projects, 
milestones and bug and issue tracking, Atlassian JIRA” was used. Tasks, feature 
requests and issues were collected in JIRA and assigned to projects and milestones. 
Open tasks and issues were discussed in the technical meetings. Overall, our JIRA 
experience was quite positive, as it enabled a quick overview of the progress and 
future work for developers and managers. Clearly, to have a consistent and up-to- 
date overview, developers have to be committed to report their work in JIRA. 

Developer camps: During the project period, three developer camps were 
organised. Originally, the developer camps were meant for internal developers to 
discuss the overall ROLE architecture and technical solutions, and plan the pro- 
jects. During the first developer camp, a shared vision of the ROLE objectives was 
created. At the second and third developer camp, external experts were invited to 
provide feedback on the architecture, identify missing use cases and requirements 
and provide a broader scope on recent research results that could be applied in 
ROLE. At the third developer camp (November 2011), we invited a larger group of 
experts, presented the current status of the ROLE framework and had a small 
developer competition to develop widgets for the platform. This was only possible 
at this time, because the implementation was mature enough. This developer 
competition was good both for dissemination to research and open-source projects 
(e.g. Apache RAVE), and for getting feedback from external developers on the 
ROLE APIs and documentation. Later we organised four more widget competitions 
that were open to the public. In general, the developer camps were a good platform 
to collaborate with the whole ROLE technical team and external experts for a 
couple of days. 


Contributing ROLE Software to Open-Source Projects 


Several components and specifications of the ROLE framework were integrated in 
other open-source projects. This strategy enables further uptake and development of 
the research results of the ROLE project. This section highlights some of the 
contributions to the open-source community. 

OpenSocial and Apache Shindig: In order to standardise the OpenSocial space 
extension (see section ‘Spaces’), EPFL worked with the OpenSocial community for 
the specification and with the Apache Shindig community for the reference imple- 
mentation of the specification. The communication with both communities happens 
through mailing lists. After our specification proposal was presented on the mailing 
list, it received very positive feedback and representatives of several companies 
showed interest in the extension for use in their products. After several discussions 
and refinements of the proposal, the work on the specification draft started. Typi- 
cally, the procedure to get a proposal accepted is as follows: First, a patch to the 


22 Atlassian JIRA, http://www.atlassian.com/software/jira 
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OpenSocial specification has to be written. Second, the proposal has to be 
implemented in an open source, publicly available platform, e.g. Apache Shindig. 
Finally, when the proposal is finalised, the community votes on the final inclusion 
of the draft into the specification. Consequently, ROLE wrote a patch for the 
OpenSocial specification and extended Apache Shindig, which was shared with 
the OpenSocial community. 

When we started our proposal, the OpenSocial community was finalising 
OpenSocial version 2.0. Thus, initially our proposal would be incorporated in the 
next version, 2.5. However, later the decision of the community was to have only 
limited changes in 2.5 and leave all larger revisions for the upcoming version 3.0. 
Hence, due to the large changes that our proposal would cause, it was decided to 
postpone its inclusion and it was only incubated in OpenSocial 2.5. Because of 
other changes in the specification of OpenSocial 3.0, our proposal had to be 
adapted. Eventually, the process that seemed open and efficient turned out to be 
quite time-consuming. Currently (December, 2013), the proposal is still on the road 
map for the OpenSocial 3.0. The patch for the specification is ready and the code for 
Apache Shindig is available. Once the work on the OpenSocial 3.0 is started, the 
patch should be evaluated and voted upon final inclusion into the newest version of 
the specification. 

However, adding a proposal into the specification does not immediately guar- 
antee that it can be used in all OpenSocial platforms. To be able to use the space 
proposal in widgets and to enable interoperability with other OpenSocial platforms, 
all platforms have to implement the latest version of the specification. There can be 
latency, since it takes time to upgrade to newer versions of OpenSocial. 

Apache Rave: As mentioned in the introduction section, Apache RAVE is an 
open-source mash-up platform with similar functionality as the ROLE framework. 
Therefore, it was a very interesting project to contribute to. Technical ROLE 
partners have joined two Apache RAVE Hackathons in the Netherlands to present 
our work and discuss collaboration. The RAVE community received our concepts 
and implementation enthusiastically. Their main interest was in our IWC compo- 
nent, the space concept and our Linked Data-based APIs to retrieve and store data in 
the PLE. As the space specification proposal was already submitted to the 
OpenSocial community, ROLE decided to propose the two other components to 
the RAVE community. The process to achieve this is quite similar to the 
OpenSocial procedure. One has to announce the idea on the public mailing list of 
Apache Rave, where the idea and its specification can be openly discussed. The 
next phase is to provide an implementation of the component in RAVE and submit a 
patch. This patch will be reviewed by the community and after acceptance can be 
included in upcoming milestones. At the time of writing, both proposals have not 
yet been accepted. We hope to get approval of the RAVE community in the near 
future. 
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Strophe.js: The parts of the IWC component have been contributed to the open- 
source JavaScript XMPP library, named Strophe.js.~° We mainly contributed our 
implementation of the XMPP protocol over WebSockets.™* This makes the library 
more efficient as data can be efficiently pushed from server to client and long 
polling is no longer necessary. 


Discussion and Conclusion 


This chapter presented the architecture of the ROLE framework and the platforms 
where this framework has been integrated. The ROLE framework provides several 
components to enable responsive open PLEs, such as IWC, automated user activity 
tracking, collaborative spaces and authentication and authorisation services to 
protect data. These components provide the basis for real-time communication 
between widgets and users and automatic user activity tracking from tools and 
services. To evaluate the usability and usefulness of the ROLE philosophy and the 
ROLE framework, we have integrated the ROLE framework in various platforms, 
such as Moodle, CLIX (Govaerts et al. 2011; Rensing et al. 2013), Graasp and the 
ROLE SDK. These platforms have been used in various real-world evaluation 
settings (Govaerts et al. 2011), which have been documented in Chaps. 4, 5, 6 
and 7. 

In general, we can conclude that with the ROLE framework we were able to 
meet the project requirements and support the test beds. The birth of the Apache 
RAVE project with very similar goals indicates the interest and usefulness of the 
ROLE philosophy. Furthermore, the framework produced several components that 
were of interest to other open-source projects. Some of these open-source contri- 
butions have been completed, while others are still in progress. Additionally, results 
of the ROLE framework will be reused and extended in other research projects. For 
instance, the ROLE Widget Store will be reused in the Go-Lab project” as a 
repository of apps and online laboratories to enable teachers to assemble learning 
environments with online laboratories for inquiry-based learning. Additionally, 
Go-Lab will also use Graasp and the OpenSocial Spaces specification to enable 
inquiry-based learning spaces for STEM education at school. On the other hand, 
researchers of the Learning Layers project”° are using and extending the ROLE 
SDK as their learning platform (Kovachev et al. 2013). As mentioned, the ROLE 
SDK is mainly meant for developers to extend their existing learning environments 


23 Strophe.js, http://strophe.im/strophejs/ 

24 WebSocket, http://www.websocket.org/ 
5 Go-Lab, http://www.go-lab-project.eu/ 

?6 Learning Layers, http://learning-layers.eu/ 
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or extend the ROLE SDK itself to support their requirements. To support this, 
developers can easily contribute or fork the ROLE SDK GitHub repository.” We 
hope that in this way large parts of our efforts will be used beyond the end of the 
ROLE project. 
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Commentary 


The following sections offer the comments of experts outside of the ROLE project 
consortium about the contents of this book. Each expert was asked to review and 
comment upon a chapter of this book that is relevant to their expertise, thus offering 
their feedback about a certain aspect of the ROLE research outcomes. 


Personal Learning Environments, Self-Directed Learning 
and Context 


Graham Attwell 


Research and development in learning technologies is a fast moving field. Ideas and 
trends emerge, peak and die away as attention moves to the latest new thing. At the 
time of writing MOOCs dominate the discourse. Yet the developments around 
Personal Learning Environments (PLEs) have not gone away. It could be argued 
that the development and adoption of PLEs is not so much driven the educational 
technology community but by the way people (and not just students) are using 
technology for learning in their everyday lives. 

Even when Learning Management Systems were in their prime, there was 
evidence of serious issues in their use. Teachers tended to use such environments 
as an extended file storage system; forums and discussion spaces were frequently 
under populated. In other words such systems were used for managing learning, 
rather than for learning itself. Learners expropriated and adapted consumer and 
productivity applications for their learning. Such trends became more pronounced 
with the emergence of Web 2.0 and social software. Social networking applications 
in particular, allowed the development of personal learning networks. Rather than 
go to the institutionally sanctioned LMS or VLE, learners communicated through 
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Facebook or Whats App. PLNs were not longer limited to class or course cohorts 
but encompassed wider social and learning networks. Wikipedia has emerged as a 
major open resource for learning. 

As mobile technologies have become increasingly powerful and, at least in some 
countries, internet access has become increasingly ubiquitous, learners use their 
own devices for learning and are not confined to institutional facilities. Regardless 
of trends in educational technology theory and research, learners are developing 
and using their own PLEs. 

At the same time, the ongoing rapid developments in technologies are changing 
forms of knowledge development and leading to pressures for lifelong learning. 
Universities and educational institutions can no longer preserve a monopoly on 
knowledge. Notwithstanding their continuing hold on accreditation, institutions are 
no longer the only providers of learning, a move seen in the heart-searching by 
universities as to their mission and role. 

Such changes are reflected in the growing movement towards open learning, be 
it in the form of MOOCs or in the increasing availability of Open Educational 
Resources. The popularity of MOOCs has revealed a vast pent up demand for 
learning and at least in the form of the c-MOOCs has speeded the adoption of PLEs. 
MOOCs are in their infancy and we can expect the rapid emergence of other forms 
of open learning or open education in the next few years. 

Learning is becoming multi-episodic, with people moving in and out of courses 
and programmes. More importantly the forms and sources of learning are increas- 
ingly varied with people combining participation in face-to-face courses, online and 
blended learning programmes and self-directed and peer supported learning using 
different Internet technologies. 

These changes are reflected in discussion over pedagogy and digital literacies. It 
is no longer enough to be computer literate. Learners need to be able to direct and 
manage their own learning, formal and informal, regardless of form and source. In 
conjunction with More Knowledge Others (Vygotsky 1978) they need to scaffold 
their own learning and to develop a personal knowledge base. At the same time as 
the dominance of official accreditation wanes, they need to be able to record and 
present their learning achievement. PLEs are merely tools to allow this to happen. 

All this leads to the issue of the role of educational technology researchers and 
developers. In research terms we need to understand more not just about how 
people use technology or learning but how they construct a personal knowledge 
base, how they access different resources for learning, including people and how 
knowledge is exchanged and developed. 

At a development level, there is little point in trying to develop a new PLE to 
replace the VLE. Instead we need to provide flexible tools, which can enhance 
existing technologies and learning provision, be it formal courses and curricula or 
informal learning in the workplace or in the community. It can be argued that while 
most educational technology development has focused on supporting learners 
already engaged in educational programmes and institutions, the major potential 
of technology and particularly of PLEs is for the majority of people not enrolled on 
formal educational programmes. Not all workplaces or for that matter communities 
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offer a rich environment or learning. Yet there is vast untapped potential in such 
environments, particularly for the development and sharing of the tacit knowledge 
and work process knowledge required in many tasks and occupations. PLE tools 
can help people learning in formal and informal contexts, scaffold their learning 
and develop a personal learning knowledge base or portfolio. 

At both pedagogic and technical levels, context provides a major challenge. 
While mobile technologies recognize the context of place (through GPS), other and 
perhaps more important aspects of context are less well supported. This includes 
time—how is what I learned at one time linked to something I learned later? It 
includes purpose—why am I trying to learn something? It includes the physical 
environment around me, including people. And of course it includes the social and 
semantic links between places, environments, people and objects. 

The challenge is to develop flexible applications and tools to enhance peoples’ 
PLEs and which can recognize context, can support people in scaffolding their 
learning and develop their own Personal Learning Networks and enhance their 
ability to direct their own learning and the learning of their peers. 

Two major European funded projects, ROLE and Learning Layers are 
attempting to develop such applications. They both have the potential to make 
major inroads into the challenges outlined in this short chapter. 
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Supporting Self-Regulated Learning 
Margit Pohl 


Current educational theories emphasise the importance of autonomous learning. 
Self-regulated learning is one example for such a theory. In the context of this 
theory, metacognition and cognitive strategies play a significant role. One of the 
goals of the ROLE project was to support metacognition and reflection of learners 
specifically. Chapter 2 on “Supporting Self-Regulated Learning” describes the 
basic ideas of this approach and its implementation in the project. 
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One relevant issue in this context is the question how much support learners need 
and how to give appropriate feedback to them. It is well known that autonomous 
learning often overwhelms learners and increases drop-out. Still, there are individual 
differences related to the ability to learn autonomously. Students with high 
metacognitive skills and self-efficacy are better able to plan their learning processes 
and learn more efficiently. The ability to structure one’s own learning and to reflect on 
the issues raised in the learning material apparently does not come naturally and has to 
be communicated to the students. In contrast to some constructivist approaches, self- 
regulated learning takes these issues into consideration. In the context of the ROLE 
project, specific assistance is given to the learners to increase metacognition and 
reflection. There is an adaptive mechanism in the support strategies which adjusts 
the learning material provided to the students to their specific needs. This mechanism 
also takes care of the fact that some students need more scaffolding than others. 

The support process is based on an extremely sophisticated framework 
consisting of a process model, a competence model and a learner model. This 
framework enables the system to give highly differentiated feedback to the learners 
without having to resort to AI methods. The framework enables the researcher to 
come up with relevant guidelines for the development and adoption of learning 
resources. In my opinion, the fact that the framework used as a basis for the 
development process is strongly related to the guidelines is an indication of the 
value of the didactic approach used in this project. In many e-learning projects the 
relationship between the underlying theory and the actual design is only very loose 
which results in a certain arbitrariness of the design. 

One of the strengths of the approach adopted in the ROLE project is that the authors 
also clarify challenges and limitations of their work. They conducted a survey with 
teachers, and they collected data at summerschools and conference workshops. These 
data indicate that the approach has advantages and limitations. Teachers described that 
advantages might be better learning from the students, more autonomy for the students 
and peer collaboration. They also see problems as, for example, the fact that many 
students are not equipped for self-regulated learning and reluctant to accept new 
methods of teaching. There are also barriers because of the way how universities or 
other educational institutions are organized. These problems have also been described 
in the literature (Laurillard 1993). The character of assessments at universities, for 
example, does not encourage self-regulated learning or reflection or collaboration. In 
addition, metacognitive skills are often not taught in schools or universities. The 
development of curricula for schools and universities is usually a highly contested 
area, and many different stakeholders try to influence this process. The introduction of 
more autonomous and self-regulated learning is, therefore, quite a challenging pro- 
cess. Projects like ROLE can play an important role in this context to present an 
exemplary realization of self-regulated learning. 

I think there are many interesting areas for future work posed by this project. 
Although some evaluations have already been conducted, a more detailed study of 
student’s interaction with the system would be very interesting. There is a pro- 
nounced emphasis on meta-cognitive activities of the students. Students have to tag 
widgets or formulate their learning goals. I think it is an interesting research 
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question how students adopt these activities. Nowadays, students are not rewarded 
for this kind of meta-cognitive activities at universities. Therefore, they might see it 
as an obstacle to get a certificate. I assume that meta-cognitive activities have to be 
integrated into courses and be rewarded in the same sense as other kinds of learning 
activities, but these are open questions which have to be investigated. 

In general, I think it would be interesting to investigate how students interact with 
this system. Choosing learning widgets and integrating their contents to form a 
coherent mental model is certainly a demanding task for the students. It would be 
very interesting to know how students cope with this task and what can be learned for 
the design of similar systems. I think that the approach using widgets which can be 
reused and combined flexibly is very promising, but it is also challenging because it is 
unusual and forces the students to reflect about their learning processes even if there 
are only few widgets to choose from. There are two aspects which I think would be 
relevant in this context. On the one hand, there is the investigation of the interactions 
and learning processes of the students. On the other hand, it would also be interesting 
to find out what kind of design can support students best in such systems. The first 
question is more didactic, whereas the second question also addresses usability issues. 

Self-regulated learning is an interesting approach because it combines a more 
active role of the learner with fairly rigorous learning strategies. Such learning 
strategies can be an advantage if supported appropriately. The ROLE project is an 
important step to implement a system to encourage a good balance between 
freedom and guidance in the learning process. 
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Multidimensional Evaluation Framework for PLE: Does It 
Make Sense and Do We Need It? 


Carlo Giovannella 


PLEs are a typical expression of our time, a time dominated by the liquidity, that 
from one side is a symptom of a profound crisis of values (Bauman 2000) while on 
the other may represent a great opportunity (Giovannella 2009), provided you are 
equipped with the skills needed to manage complexity. PLEs are virtual environ- 
ments in continuous evolution, potentially no-places (Augé 1992) without memory, 
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containing the promise for highly customizable environment and learning processes 
as amply illustrated by this dedicated publication. 

PLEs, thus, are not suitable for most of the today’s learning processes and their 
actors. Certainly not for present teachers, who do not “shine” for the mastery of an 
adequate digital literacy and who, in the vast majority, still consider virtual environments 
useful as content repository or message boards. Neither for most of the students that, 
although belonging to the so called digital native (Prensky 2001) and showing a 
considerable ability/independence in managing interpersonal communication, when 
are asked to take the responsibility of their own training path step back and, actually, 
prefer to be hetero-directed and evaluated by teachers. PLEs, thus, are not for today, but 
represent an interesting laboratory within which one can experiment around the central- 
ity of the person and her ability to design her own learning trajectory according to design 
based learning approach, her ability to acquire suitable design literacy (Giovannella 
2010) and other twenty-first century skills (Giovannella and Baraniello 2013). 

A smooth introduction of widgets usage into more “traditional” learning envi- 
ronments would be, thus, largely advisable to foster the transition toward more self- 
regulated learning paths. 

Considering the present conditions the organizational level of the evaluation, 
although should be considered to design a general framework, is too far away with 
respect to the nowadays urgencies. Since in PLEs the PERSON and her learning 
EXPERIENCE is expected to be at the centre of learning process, the evaluation 
should focus mainly on interplay and co-evolution of the “characteristics” of both 
people and techno-ecosystem. 

As well explained by the authors of Chap. 3 one should go well beyond the 
standard HCI prescriptions to embrace the whole multidimensional spectrum of the 
human experience mediated by the machine. Of course one needs to develop a better 
understanding of the learning experience, develop meaningful models (Giovannella 
et al. 2011) and try to make these latter as robust as possible. New evaluation methods, 
thus, should be developed and integrated within (or made more easily accessible from) 
PLEs, and more in general all kinds of learning environments. 

The goal should be the multidimensional evaluation of the EXPERIENCE and, 
of course, of: 


The learner ability to design her learning process (not just to follow the proposed 
one). 

The acquisition of the relevant competences and literacies and among them the 
ability to interpret the analytics and self-evaluate her own evolution and needs. 
Accordingly the evaluation and redefinition of PLEs usage has to capitalize on 
the large and well established methodological corpus that have been developed 
in the past 20-25 years within many disciplinary domains: anthropology, psy- 
chology, sociology, computer science, interaction design, design for the experi- 
ence, design, etc., and that has been well synthesized in Chap. 3 of this book. A 
corpus that can be even enlarged to consider many other methods (the descrip- 
tion of which can be easily found on the web) and that should also be integrated 
by new approaches and methodologies suitable for the multidimensional 
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monitor of the learning experience (Giovannella et al. 2011, 2013). A task, this 
latter, that has been accomplished also by some partners of the ROLE project. 
The debate on qualitative vs. quantitative methods and subjective vs. objective 
data detection can be considered an ill posed one. No one would renounce to 
more objective data, collected in an unobtrusive and respectful of privacy 
manner, no one would renounce to push the border from qualitative toward 
quantitative data detection, when possible. The debate, thus, has better to 
concentrate on the quality of data (i.e. “smart data” instead of “big data”) and 
on the ability to interpret them. 

As an example, an apparently highly objective detection method like the 
eye-tracking when not well controlled may produce unreliable results if indi- 
vidual visualization styles are not dutifully taken into account. 

As additional example, emotions and sentiment, apart the need of well-grounded 
and interoperable models, could be both objectively and subjectively detected, 
but the choice of the approach strongly depends on the time-window of interest 
and cannot avoid to consider both emitters and detectors, whatever communi- 
cation modality (voice, text, images, etc.) and medium are involved. 


In conclusion, PLEs are learning labs challenging all actors of learning pro- 
cesses and researchers in many respects, including a person/people in place 
multidimensional monitoring to detect the acquisition of meta-design literacy, 
self-direction and self-evaluation skills. 
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PLE in Formal Education: Challenges for Openness and 
Control 


Marco Kalz 


Abstract This short comment reflects on a critical account of educational tech- 
nology and makes reference to the chapter by Vieritz et al. about the use of widget 
bundles for formal learning in higher education. 


Introduction 


Personal Learning Environments (PLEs) have been intensively discussed since the 
introduction of the concept without an agreement about their definition and con- 
crete focus. In its early development phase PLE has been introduced as learning 
technologies under the control of the learner (van Harmelen 2006). Later we have 
described the PLE as a learning environment in which learners on the one hand 
actively integrate distributed digital information, resources and contacts, on the 
other hand document learning progress and learning outcomes based on standards 
(Schaffert and Kalz 2008). While the original concept of the PLE has been 
introduced as a counter-concept for teacher/instructor-prepared learning environ- 
ments like Learning Management Systems (LMS) nowadays this perception of a 
PLE seems to have moved into a direction in which all technology that enlarges the 
landscape of standard learning technology can be regarded as a PLE. 

The authors of the chapter have presented three case studies of widget bundles 
that function as an enrichment of the traditional technology-supported learning 
environments at these three institutions. These implementations provide interesting 
directions for a transition between learning technologies that are designed 
according to fixed curricula and prepared content towards more flexible environ- 
ments. Especially the activity recommender might offer an interesting direction to 
support self-organized learning. But flexibility alone is not the core of a PLE. 

Selwyn calls for a critical account of educational technology that takes into 
account the societal intertwining of educational technology on the micro-and 
macro-level and the study of learning technology in dimensions of “power, control, 
conflict and resistance” (Selwyn 2010). We cannot disconnect this wider discussion 
and reflection from the implementation level. In this sense, learners need to be able 
to actively (co-)design their learning environment to make it a personal one. This is 
the important difference between adaptivity and adaptability of a learning environ- 
ment (Oppermann and Rasher 1997). While adaptivity can be designed completely 
according to rules of teachers or the designer of a piece of learning technology, the 
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adaptability enables a learner to design the learning environment according to 
individual needs. In the context of educational institutions and formal learning 
this leads to a number of challenges. 

The authors have argued that pre-designed widget bundles have been used to not 
confuse users and provide them with too many choices. But this leads to the 
contradiction that widget bundles are a result of a design process of teachers 
without giving learners any influence on their technology-enhanced learning envi- 
ronment. We have described this contradiction as a “competence continuum” 
consisting of a number of core skills to be able to use a PLE effectively for self- 
directed learning (Wild et al. 2009). The biggest challenge is to come to a setup that 
also enables learners without a high level of self-directedness and IT skills to slowly 
get used to a more open and flexible learning environment. Pre-defined spaces that 
can slowly be extended are one option for this issue, the other option would be to 
make available a limited number of widgets that users try first and then decide about 
their use and usefulness. 

And this leads to a related challenge: Since PLE are dynamic environments that 
grow according to the context and needs of the learner their evaluation needs to take 
into account a temporal perspective consisting of a number of snapshots of the 
environment and their impact on enabling self-directed learning processes. It is 
essential for the further development of PLE and their impact in education that the 
community develops evaluation frameworks that can systematically handle the 
complexity of evaluating a personal environment that changes its status dynami- 
cally over time and can thus fulfill different purposes. 

One possible theoretical framework for developing such an evaluation approach 
is the adaptive structuration theory: ,,The act of bringing the rules and resources 
from an advanced information technology or other structural source into action is 
termed structuration. Structuration is the process by which social structures (what- 
ever their sources) are produced and reproduced in social life” (DeSanctis and 
Poole 1994). Thus can this theory build a good foundation to analyse the interre- 
lation between social structures and technological structures developed in a PLE 
and the dimensions pinpointed by Selwyn. 
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The Future of PLEs: How Can Higher Education Be Passed? 


Martin Ebner 


Woolly Thoughts on PLEs for Higher Education 


I just read Chap. 5 on “Case study 2: Designing PLE for Higher Education” and 
would like to sort my thoughts. On the one side the ROLE (Gillet et al. 2010) as 
well as the Go-Lab project (Gillet et al. 2010) took us a step forward to see how the 
future of teaching and learning might look like, on the other side we ourselves at 
Graz University of Technology also gathered experiences how a PLE is used in 
Higher Education (Ebner et al. 2011; Taraghi et al. 2010). From this personal 
perspective I would like to enhance the chapter bringing three dimensions in 
mind. Three factors have to be considered when introducing a PLE to Higher 
Education institution, at least in middle-Europe: 


1. Technological perspective: First of all as already written in the chapter a 
Personal Learning Environment offers more or less both—freedom and restric- 
tion. Learners must be able to choose their personal applications, contents, tools 
for their individual learning process, but should be also able to do this in a secure 
and private way. In contrast to a teacher-centred Learning Management System 
we are talking now about a user-centred, flexible, expandable system. From a 


M. Ebner (%4) 

Department Social Learning, Computer and Information Services, Graz University of 
Technology, Graz, Austria 

e-mail: martin.ebner@tugraz.at 


Commentary 229 


technological point of view it is a kind of a multi-application monitoring 
environment according to the special needs of a specific learner. 


To achieve this goal those platforms are following a widget-based MashUp 
concept (Taraghi et al. 2011) where different small applications (widgets) can be 
arranged by users themselves. The web-based software consists mainly of two 
parts—a framework (the widget container) and the widgets themselves. So the 
weakness of the concept is maybe also its strength—to run such an environment a 
high number of widgets for different purposes or learning goals are needed. Graz 
University of Technology follows the concept of users’ programmed widgets, 
which means that students of informatics are doing this small applications during 
their projects or exercises (Taraghi and Ebner 2010). 


2. Organizational perspective: The second major factor of a PLE in Higher Edu- 
cation is the question who is running such an environment and what does that 
mean to our lecturers? On the one side it seems rather obvious that the system 
has to be provided university-wide on the other side it must be brought into the 
mind of each single user—lecturers as well as students. First experiences pointed 
out that in general such an environment is intuitive and can be well explained 
with the “App-store metaphor”. Due to the fact that nearly everyone owns a 
smartphone today it is easily imaginable if a Widget is called App and the 
Widgetstore is compared with the App-store. First gathered statistics pointed 
out that the PLE in general is used if it is provided university-wide, but still more 
or less for getting-information issues than teaching and learning purposes 
(Selver et al. 2013; Taraghi et al. 2013). 

3. Teaching and learning perspective: Finally it must be taken into account that any 
system for supporting learning and teaching needs a certain context where it is 
used and an embedded didactical scenario (Ebner et al. 2011). As described well 
in the chapter using a PLE for teaching and learning will be a switch from 
behaviourism to cognitivism. Most of our daily lectures in typical bachelor 
programmes are based on a face-to-face education where lectures present their 
contents. It is obvious that this kind of teaching is not appropriate for such an 
environment where students should aggregate, share, search, recommend etc. It 
can be summarized that an arbitrarily effort will be necessary on this issue. 


Future of Higher Education will need therefore new concepts, lecturers who 
revise their lectures and learners who will adapt their learning styles. The concept 
of a Personal Learning Environment and its technical realization is just a first step 
and the chapter as well as the whole book a first great tribute to it. 
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Exploring Open Educational Resources for Informal 
Learning 


Jon Dron 


I have been following the ROLE project since its early days and I am delighted to 
read this report of some of the resulting insights and ideas about how lifelong 
learning may be supported with its tools. I would like to take this opportunity to 
interpret some of the findings in this chapter, drawing on both the chapter and my 
own experiences with the development of widget-based PLE tools. 
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One of the most interesting findings related in this chapter is the mixture of 
feelings expressed by participants as to whether the PLE would be useful. For 
instance, one of the responses from workshop 2 suggests that the effort needed 
might not be worth the trouble and, more explicitly, one participant in the Dev8eD 
workshop comments on the potential difficulties of integrating the ROLE widgets 
with their existing PLE that included EverNote, Twitter and Google calendar. PLEs 
are not filling an empty niche: we all assemble our own PLEs, whether we call them 
that or not. At a broad level, there are PLEs that seek a high level of integration and 
management of disparate learning tools (that I will refer to as iPLEs), and others 
that are mostly an aggregation of tools (that I will call aPLEs). The ROLE tools fall 
mainly into the iPLE category: their purpose is largely to make it easier to aggregate 
and integrate learning spaces and resources. If this is to work, then it must have 
extra value not found in other parts of an aPLE. It must be worth the effort to learn 
to use them. My own aPLE includes a range of personal and shared aggregation 
tools like browser bookmarks, Pocket, RSS readers and EverNote; productivity 
tools like calendars, email, Google Search and Apple Widgets; learning objects 
everywhere, from Wikipedia pages to StackOverflow answers; telephone, Skype, 
Adobe Connect, social networks, Google Hangouts for dialogue; shelves of phys- 
ical books as well as virtual collections; a desk, a range of computers and mobile 
devices; and, most significantly, a set of methods, procedures and pedagogies from 
which I choose to assist my learning process. Altogether, it is a flexible, highly 
customized personal learning environment that I use to assemble the things I need 
for my own learning. There would need to be a good reason to add more tools to this 
mix. This leads to another quite closely related major issue raised in the chapter: 
that of usability. 

The chapter highlights issues of usability and technical complexity. This is a 
wicked problem because PLEs tread a tightrope. They must provide a lot of 
flexibility in order to support an indefinitely large number of potential self-guided 
learning strategies but they must also make learning easier. For flexibility, they 
must be fairly soft technologies, in which orchestration of processes and methods is 
performed by their users. Unfortunately, the softer we make our technologies the 
harder they are to use, because we must put in the effort to perform the orchestra- 
tion. If we harden our toolset then some parts of the orchestration must be handled 
by the tools but, the more orchestration that is built into a technology, the less 
flexible it becomes. Efficient, demanding less thought, fast: but rigid. Widgets offer 
a potential solution, by allowing small hard pieces to be assembled into a vast range 
of learning environments. Using any single widget is mostly pretty simple but 
knowing which widgets to choose, how they can be configured, how they can be 
arranged and what they can be used for is much more complex. Thus, though the 
pieces may be relatively hard, the overall assembly remains soft and therefore 
difficult to use effectively, requiring an investment in learning and configuring that, 
unless proven worthwhile, is unlikely to be attempted. 

When we talk of self-guided learning we normally mean it only at a coarse 
granularity: essentially, the absence of an overarching course structure. At a smaller 
scale, structured learning objects, book chapters, websites, videos and many other 
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teacher-created artefacts are the norm (using “teacher” to mean anyone, including a 
team of designers or fellow learners, that intentionally or otherwise helps another to 
learn). So it is with interest that I read this chapter reporting on personal learning 
environments, but talking about them in the context of intentional teaching, 
courses, workshops and other planned processes. Self-regulation can occur at 
many scales. We may choose to control different aspects of the learning process 
but almost always delegate control to others at many stages, whether to the author 
of a chapter or learning object, the leader of a workshop, our PLE or the widgets 
within it. 

Some tools described in this chapter such as Etherpad and Flashmeeting hinge 
on social engagement, which entails a need to be at the very least mindful of the 
schedules, needs and goals of others. This highlights a tension that exists in nearly 
all PLE implementations, that they support our social learning activities, but that 
those social learning activities themselves, with our fellow learners and teachers, 
provide shape and form to our learning. For instance, I was not surprised to read that 
relatively little use was made of Etherpad and Chat in the events described: given 
that participants were collocated it would not normally be very useful to provide 
alternative real-time collaboration tools, especially as the tasks did not appear to 
focus on production of a permanent artefact but were simple part of some active 
experimentation to use the toolset. 

At the heart of all my reflections on this chapter is the fact that PLEs are more 
than just a way to keep things organized in our learning lives. Done well, they are 
generative toolsets that can act in some ways like a teacher, offering guidance, 
inspiration, motivational support and structure to the learning experience. But, at 
the same time, they seek to provide freedom from such a teacher role, to be soft 
tools to support self-regulated learning. They are thus both teachers and not 
teachers at the same time. Their innate softness is perhaps the reason that the 
evaluations performed in this chapter focused on helping people to use the tools 
in a manner that is anything but self-regulated and explains why it is so hard to pin 
them down. A PLE is personal: every individual builds processes and methods 
around them, configures his or her own space but, at the same time, that space is 
shaped and influenced by the people, resources, learning objects, tools and expertise 
that are available. This tension lies at the heart of education. When we educate 
ourselves we choose the parts that we delegate to others more than those who follow 
a more guided path but, through the shape of our tools, the people around us and 
simple path dependencies, we have many of our decisions made for us and, at a finer 
granularity, always delegate at least some of the teaching process to others. Getting 
the right balance is a tough task to perform well and partly explains why case 
studies like the one presented here have a vital role to perform in helping us to 
understand that better. 
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Workplace Learning at Festo: Content Creation by Workers 
for Workers 


Content Creation by Workers for Workers 
Juergen Mangler 


Abstract As a follow-up to Chap. 7—an interview with Manuel Schmidt and 
Michael Werkle was conducted, to delve into how it is possible for Festo to 
embrace PLE concepts in their organization, about the motivation of workers to 
create and share content, and about future projects. 


Keywords Interview e PLE e Content creation 


Introduction 


From Chap. 7 we learned that FESTO is to promote informal learning—learning 
that happens e.g. through interaction of the learners in social media environments. 
Rather than employing a top down approach, where learners are fed learning 
material prepared by dedicated content creators, an individualization of learning 
is aspired for that covers the learning process (i.e. the what, when and the pace), as 
well as the content creation process. For the content creation process FESTO is 
focusing on the experts in the fields (i.e. the workers who do something day to day). 

In order to find out more about how typically strict learning related policies in a 
company fit in with the goals of PLEs, the interview was conducted with a focus on 
the philosophy and benefits related to letting workers create their own learning 
material during their work-time, and how this fosters learning and collaboration in 
the workplace. 

Both interviewees are members of Festo Lernzentrum, a separate entity inside 
Festo that maintains the companies’ Learning Management System (LMS), oversees 
content creation, as well as the creation of didactic designs for the training of workers. 


J. Mangler (>) 
Faculty of Computer Science, University of Vienna, Vienna, Austria 
e-mail: Juergen.mangler@univie.ac.at 


234 Commentary 


Interview 


Interviewer: What is the relationship between traditional e-Learning and PLE’s in 
your Organization, and how was it shaped by the ROLE project? 

Manuel Schmidt: The ROLE project served as an incubator for complementing 
the existing LMS with properties of a Personal Learning Environment (PLE). From 
the start of the project there was a strong consensus inside Festo, that a pure PLE 
was not suitable for the business context. As motivating workers to maintain and 
extend their job related skills and knowledge is a primary focus of businesses in 
general, they also want to control the goals and granularity (i.e. the when and what) 
of learning. Furthermore businesses want to speed up the process of getting started, 
which they see as conflicting with the nature of PLE’s, where the user starts from 
scratch, e.g. spending lots of time building one’s own learning environment through 
widgets. 

Interviewer: Can you describe the typical learning scenarios for Festo workers? 

Manuel Schmidt: Festo is, like possibly most businesses, very much focused on 
individual workers’ career paths and individual competence development plans. 
This formal training of workers is accompanied by e-learning. We focused on 
allowing workers that have to tutor other workers to create and distribute their 
own content, e.g. videos. (...) 90% of our users are knowledge workers. 

While all workers of course are actively encouraged to participate in seminars, 
which are rigidly structured, we provide complementary e-learning content in our 
LMS. For this content, workers can decide for themselves if and which parts they 
want to consume. So even if a learner does not participate in seminars, he/she can 
select from a wide array of on-line learning material. 

Interviewer: Do you have an estimation which percentage of users use your LMS 
to consume content because of—or complementary to—seminars, and which per- 
centage of users are purely self-motivated learners? 

Manuel Schmidt: During 2013, about 50 blended learning seminars with oblig- 
atory material provided through the LMS took place, but about 600 courses in total 
have consumed during the same year. 

Michael Werkle: Staff development in Festo relies on two pillars—quantitative 
and qualitative goal-setting between workers and their supervisors, and self- 
motivation. The facts are: the 600 courses had a total of about 8,000 users consum- 
ing them, and learning videos have been consumed over 13,000 times in the 
last year. 

Interviewer: In Chap. 7, section ‘Implementation of the Personal Learning 
Management System’ it is stated that one organizational requirement is to ensure 
the transparency of the learning process and the yielded achievements. For sure the 
employer is interested in, and encourages its workers, to improve their skills and 
competencies. Does Festo have any formal instruments in place for motivating 
workers, for example awards? 

Manuel Schmidt: There are no awards for learners or content creators. We were 
toying with the thought, but so far nothing has been realized. One important aspect 
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in our organization is the very open culture regarding learning: we allow and 
encourage the learners to use our on-line resources during normal work-time. 
This trust vis-a-vis the workers alone translates into motivation. 

Michael Werkle: Rewards for content creation can of course also have negative 
effects. For example the use of Wiki’s in companies is often very successful when 
coupled with reward systems ... but they immediately die as soon as the rewards 
are taken away. The learning-tube philosophy is successful without any extrinsic 
rewards. 

Manuel Schmidt: We think that systems are successful when the intrinsic reward 
is obvious for the workers. For example, knowledge workers that have to train other 
workers are much more flexible when they create videos. The can reuse them in 
seminars or even refer to them during normal meetings. 

Interviewer: Do you track individual learners, their learning progress, .. .? 

Michael Werkle: This is not possible due to German privacy protection laws, and 
company level agreements. 

Interviewer: Currently the whole approach seems much centred on knowledge 
workers—workers who do all their work in front of the computer screen—or at least 
on the knowledge working part of the job. Is there a planned integration of the 
factory floor learning processes into the system? For example when a worker 
explains to other workers how certain systems on the factory floor work, or how 
to make them work better, it cannot be captured by screen-casts. 

Michael Werkle: There are several research projects underway for human- 
machine interaction. We are not sure yet which direction these projects will take. 

Manuel Schmidt: For me the question is—how to integrate the LMS into the 
work environment—or maybe not integrate it at all. For example, content could be 
attached to machines, and a learner equipped with technology like ‘google glass’ 
could access this content directly in front of the machine, in a augmented reality 
setting. The goal will be ‘integration into the normal work environment’. 

Interviewer: A very simple step, long before producing content for others is: 
taking notes for yourself in order to not forget. Learning material is produced for 
self-consumption. Are there any signs that the learning facilities inside Festo are 
used like this? To what extent? 

Manuel Schmidt: Our content creation process (for learning-tube, Ed.) is 
two-tiered. First the created content is saved locally, and only in a second step it 
is published. I know that some colleagues are using the system for personal notes. 

My personal estimation is that the ratio between published videos and local 
videos—consisting of videos that the users are not happy with, and videos that the 
users created for self-consumption—is about 1:10. 

Interviewer: In the conclusion of Chap. 7, it is mentioned that the search 
mechanisms inside videos are not yet there—specifically full text search is not 
working because not automatic translation of the spoken word to text is possible. 
Are there any new developments in this area? 

Manuel Schmidt: We tried to set up a project that tackles full-text search for 
videos with a semi-automatic approach: machine-translation and human lectors. 
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We furthermore wanted to create tag-clouds for each video. Currently our search 
only uses tags that have been added by the creator of the video... 

Interviewer: ... but are the learners allowed to add own subtitles, tags, and notes 
to learning videos—basically crowd sourcing the creation of data for full-text 
search? Semantically conceptualized information is after all much more valuable 
than just the plain full-text, as it allows to find something according to its meaning, 
instead of the words that are used by the creator. 

Manuel Schmidt: This is currently not possible, but will be added in the future. 
For now we focused version of the platform that supports a recommender system 
for videos, including comments. 

Michael Werkle: Especially interesting—and related to this topic—is interna- 
tionalization. As we are a multi-national organization, we observe the workers— 
e.g. colleagues from US and Germany collaborate for bi-lingual videos. It is 
especially important that the tools not only support such collaboration, but make 
it easy. 

Interviewer: As mentioned in the introduction, media-didactic conception and 
design is key to the success of learning material. How does Festo tackle the fact that 
when the content is produced by experts in the field (i.e. with no extra media- 
didactic education)? Is there a support team that helps the workers who are willing 
to produce content, without putting an additional post-production burden on him? 
Or is the content left unaltered and filtered purely by how well is received by other 
learners? 

Michael Werkle: Our observation is: the quality is very high. The content 
creators are aware that about 15,000 colleagues—including the upper 
management—can watch created content. Thus they put lots of efforts into the 
created material. Usually we only have to provide technical support regarding the 
tool—and more generic tips, like how to best present my desktop. The users 
definitely put lots of effort regarding message and scenario into the content creation 
process. The users even come up with lots of ideas for the presentation of topics that 
we would have not thought about. 

Interviewer: This raises the question about granularity? How is ensured that one 
video does not contain too much information (that could be split up into smaller 
pieces—micro-learning)? 

Manuel Schmidt: Videos typically have a duration of 2—10 min, so the content 
creators intuitively go for the right granularity. As most created content deals with 
solving a specific problem, a storyline is natural: explain the problem, solve the 
problem, happy end. 

Also one big group of content creators is definitely key (region, E.d.) managers 
and product managers—they know how to sell products and thus are also qualified 
to create learning material. 

On the other hand, also people with no special skill-set, which even could be 
described as introverted, created content. In some cases these people work together 
on videos, i.e. interact regarding the topic of the video in the form of a question- 
response game. In this case I suspect one person alone would not have created a 
video. 
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Interviewer: Evaluation is always a time-consuming process that is easily 
criticisable because of e.g. sample size, or certain questions used. Additionally, 
the quality of the properties of a tool is distorted by the very content it provides—in 
the case of Festo, the videos produced by workers. But they also only produce a 
snapshot—a glimpse into what a limited group of people thought at a certain time. 
Is there a permanent evaluation mechanism, that continuously and preferable 
without placing additional burden on the users (a passive mechanism)? Quantity 
is important, but what about quality? 

Manuel Schmidt: We differentiate between LMS and content. We have a con- 
tinuous evaluation regarding learning-content, the platform—its functionality—is 
evaluated with each new release. Currently questionnaires are created in conjunc- 
tion with courses; the functionality for evaluating single learning objects like videos 
is included in our next internal release. 

Interviewer: Thank you for the interview. 


Conclusion 


As a conclusion from the interview, it becomes apparent, under the premise that a 
learning-friendly culture has been established, that businesses have no problems 
finding motivated workers that create high-quality content, and cooperate with 
colleagues in content creation. Finding a balance between pure PLE’s and tradi- 
tional LMS seems to be a bit of struggle. Due to already existing formal educational 
instruments like seminars, and time-restrictions, properties of PLE’s seem to be 
hard to integrate into a company strategy. At Festo the idea of PLE’s is manifested 
as a comprehensive library of learning objects—videos, courses, material—that the 
learners can select from. It will be very interesting to observe if other companies 
will move into the direction of allowing learners to create and share own material 
and courses at an even more fine-grained level. 

Finally, the topic of integration the factory floor—non white-collar workers— 
into these systems will be a challenge, with lots of innovative concepts to be tried 
out. For example the idea to attach learning material to physical objects including 
the consumption in an augmented reality setting seems very intriguing. 
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Lessons Learned from the Development of the ROLE PLE 
Framework 


Sheila MacNeill 


Extending the flexibility of learning environments continues to be a challenge for 
both users and developers. Over the past decade there has been increased demand to 
ensure that institutionally provided learning spaces are easily adaptable and per- 
sonalized. Many teachers, learners and VLE administrators have been frustrated by 
the lack of flexibility and opportunities for customization and personalization in 
VLEs. Recently, there have been a number of developments that are allowing far 
more flexible and open approaches to be taken. 

In 2010, in response to the increased demands both pedagogically and techni- 
cally to integrate more social applications into VLEs, Cetis produced the Distrib- 
uted Learning Environments Briefing Paper. The paper outlined the tensions at that 
time as: 


the requirement for deeper integration with other (administrative) systems gave rise to the 
MLE (managed learning environment) concept. Later, the demand for greater 
personalisation and the availability of new web tools gave birth to the PLE (personal 
learning environment) debate, in which people radically re-conceptualised the notion of a 
learning environment. During these phases, however, the VLE still remained a dominant 
force within institutions. This has resulted in a tension between the role of the VLE as a 
common tool for the institutional community, the desire to make it permeable to the 
institutional network and the wider web and to allow greater levels of personalization/ 
customization for individuals and institutions. 


The chapter outlined five potential models for the opening up and integration of 
VLEs with a number of other administrative systems and the wider social web and 
allowing increasingly flexible access to VLEs from mobile devices 

Following the publication of the paper, Jisc funded a small development 
programme, Distributed Virtual Learning Environments’ that allowed several rel- 
atively small-scale projects to develop solutions based on the models. 

Interoperability and flexibility have, and continue to be, central to the work of 
Cetis, so as this programme developed we actively engaged with a number of other 
communities working in this space including the Apache Foundation and the ROLE 
project. 

The work of the ROLE project was of particular interest as it provided a useful 
intersection and more potential technical solution to some of the outstanding 
challenges from the programme. In particular by providing an underlying open 
architecture for the creation, deployment and storage of widgets. The areas of 


' www.jisc.ac.uk/publications/briefingpapers/2012/extending-the-learning-environment.aspx 
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development discussed in this chapter were of particular interest in terms of 
providing potential solutions to HEIs in moving forward the development of 
more flexible learning spaces. 


Interoperability Framework, Architecture 


The open framework and architecture developed through ROLE demonstrates the 
key functionality and communication mechanisms for the deployment and integra- 
tion of widgets. A significant challenge highlighted by the Jisc DVLE programme 
was that institutions do not have the capacity to host and manage institutional 
widget spaces/stores. However the concept of a central, educational specific “app 
store” did have traction. The concept of an educational specific app store had 
traction; it just needed a mechanism to make it a reality. Following the Cetis 
2012 App Store Conference Session,” Jisc funded a pilot project for the Role project 
to produce a proof of concept store utilizing their developing infrastructure and 
architecture. 


Inter-widget Communication 


For widgets to be integrated within a successful PLE, it is necessary for them to be 
able to integrate with other elements of that environment. Collaboration is an 
increasing part of many learning experiences. Widgets offer an array of customized 
collaborative activities. One of their inherent appeals is the fact that learners/ 
teachers can utilize a variety of widget combinations. In an educational setting 
such as a course delivered primarily via a VLE, widgets need to be able to access 
key user information and recognize individuals and groups. 


Contextualized Meta-data 


As the chapter highlights tracking widget interactions is central to developing 
responsive learning environments. The growing interest in learning analytics in 
the sector also points to the desire for more detailed information on user activities. 
The exploration and instantiation of the CAM schema as described highlight the 
affordances (and challenges) both for end users and developers that this method of 
data collection can provide. 


2 http://blogs.cetis.ac.uk/sheilamacneill/2012/02/26/app-stores-galore-at-cetis12/ 
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Spaces 


Learning spaces are notoriously difficult to comprehensively define. The bound- 
aries are constantly evolving and being permeated. The concept of space(s) defined 
by the ROLE framework allows the aggregation of people, resources, applications 
and spaces. Two approaches—linked data and Open Social are discussed. Both 
have their strengths and weaknesses, which have been explored and expanded 
through the work of the project. 


Authentication and Authorization 


Authentication and authorization of users and data is a vital element of widget 
deployment in a PLE context. The chapter highlights the two levels of data 
communication methods needed for authentication and authorization—service-to- 
service and widget-to-widget. More work has been done on the later, in particular 
with recognized authorization services such as OAuth, which provides a level of 
user control over sharing of data in specific spaces. 

The chapter provides a comprehensive overview of the potential of the ROLE 
framework in the development of increasingly adaptable and sophisticated personal 
learning environments. From a personal point of view, it was very rewarding to play 
a small part in joining up developments within the UK HE sector with the wider 
European context provided through the ROLE project. 
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Supplementary Material 


Links to relevant websites and software 


ROLE Project Website: 

http://www.role-project.eu 

ROLE Widget Store: 

http://www. role-widgetstore.eu 

ROLE Software on GitHub: 
http://github.com/ROLE/ROLE 

ROLE Software Documentation: 
http://role-project.sourceforge.net/wiki/index.php/Main_Page 
ROLE Network on LinkedIn (currently 720 members): 
http://www. linkedin.com/groupInvitation? gid= 1590487 
ROLE Videos: 

http://www.role-project.eu/Videos 

ROLE Online Courses: 

http://tinyurl.com/role-course 
http://tinyurl.com/role-srl-course 

ROLE eBook: 

http://bit.ly/self-regulated-learning 
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